Friday, July 25, 2008

Hyper-V on the laptop (Compaq nc6320)

I want to run Hyper-visor and other software (SQL Server 2008) on my laptop. I consider running Team Foundation Server, Windows XP, SQL Server versions, old Visual Studio versions, Visio, BizTalk and others inside Hyper-V images on my laptop. At the same time I want the laptop to function as normally (internet explorer, media player, office, etc).

Hyper-V runs on 64-bit Windows Server 2008. The machine needs to support hardware-assisted virtualization and data execution prevention, and these must be enabled in BIOS.

  1. Download driver packages for Vista x64 and save on flash-drive
  2. Install Windows Server 2008
  3. Rename computer
  4. start control userpasswords2 and enable automatic logon
  5. Install drivers and enable wireless service (not automatic on server)
    1. Install Intel Graphics sp37719
    2. Install "Integrated module with Bluetooth wireless technology for Vista" SP34275 don't care if it says "no bluetooth device"
    3. Replace wireless driver: \Intel Wireless sp38052a\v64
    4. Add feature "Wireless..."
  6. Connect to internet
  7. Start updates
    get updates for more programs
    automatic
    important updates, only
  8. View available updates
    Add optional: Hyper-V
  9. Install Hyper-V
  10. Add role Hyper-V

The machine is now a Hypervisor. Next steps are to get Windows Server 2008 to act more like a client OS.

  1. Install other drivers
    1. SoundMAX SP36683
  2. Install Feature "Desktop Experience" (Media Player, etc)
  3. Create shortcut "net start vmms" for starting Hyper-V and "net stop vmms" for stopping Hyper-V
  4. Set startup for service "Hyper-V Virtual Machine Management" to manual
  5. stop vmms
  6. Install Virtual PC 2007 (in case I want wireless network from a vm)
  7. Enter product key and Activate windows

Since I have a working version of Windows, this is the time to activate, next step would ideally be to install office. I don't have office handy (am on vacation), so I install the viewers for Word, PowerPoint and Excel

  1. Word viewer
  2. PowerPoint Viewer
  3. Excel viewer

I am now ready to install Visual Studio and the servers, first out is SQL Server 2008 rc.

  1. Set up IIS (with all the iis6 stuff)
  2. Set up Windows SharePoint Services 3.0 x64 with Service Pack 1 to run "stand-alone"
  3. (optionally) set up Windows SharePoint Services 3.0 Application Templates – 20070221
  4. Create user ServiceAccount to run all the sql server services
  5. Install SQL (complete) with RS in SharePoint integrated mode
  6. Open firewall (http://go.microsoft.com/fwlink/?LinkId=94001)
    open tcp 1433 (engine), 2383 (ssas), 80 (sp), 135 (ssis), 443 (https), 4022 (service broker)
    udp 1434 (ssas)
    and "notify me when new program blocked"

Now get all updates.

Next step is to install Visual Studio, which should be straight-forward.

OK, I'm off to the beach,
Gorm

Tuesday, July 15, 2008

CMD: redirect error output

I write quite a few batch-files these days and want the error messages to be supressed. I had a vague recollection telling me that this was possible, and found it googling, so here it is:

"2>nul".

In a sample:

Del *.log /f /q 1>nul 2>nul

Deletes all log-files, hiding output (1>nul) and error output (2>nul). If I remember correctly (3>nul) was hiding printer output. In the old days. You can, of course redirect to files as well:

Del *.txt /f /q 1>out.txt 2>err.txt

Del *.log /f /q 1>>out.txt 2>>err.txt

This sample is supposed to put all the output in out.txt and all the errors in err.txt.

Gorm

Sunday, July 13, 2008

ETL top-down 1 – Architecting abstraction layering

I work at a very exciting BI-program where we have been able to do things right. The project is part of a larger program that contains Business Consulting, Master Data Management and Business Intelligence and, who knows, maybe even Information Lifecycle Management.

My work on the project includes ETL on the Enterprise Data Warehouse, a central project in the program.

My trade is, and has for some time been architecting developer. This gives me advantages early in the ETL process when it comes to what I like to call "Abstraction Layering"

Abstraction Layering – customer perspective (why)

Abstraction layering helps to set the balance between "keeping things open" vs. "delivering as soon as possible".

For my current project we need to deliver quickly while handling a few issues (from the top of my head):

  • Loosely defined long term goal – I think
  • Distributed developers, both geographically, and experience-wise
  • Many source systems
  • Large master data management and other projects changing the environment
  • Real-time
  • Right time
  • Traceability

When implementation is done, we need to focus on simple measurable tasks. One way to do this is to model the work on well defined levels of abstraction. This way we can design top-down by having the most abstract discussions first, then some intermediate discussions and lastly the implementation details.

Things we do interesting to ETL includes:

  1. Selecting reference architecture.
    Master Data Management, Hub and spoke EDW with 2G, full archive of source systems, data marts, custom Meta Data Repository.
  2. Create ETL "horizontal" layering – interfaces and documentation.
    Packages take data from one architecture layer to another, grouping functionality and enabling measurability.
  3. Create ETL "vertical" layering – restrictions and grouping.
    Jobs uses "job packages" uses "aggregated packages" that groups functionality in measurable chunks.
  4. Specify update intervals and delivery.
    We plan for nightly job, hourly job and real-time job. Monthly, weekly reports, operational BI and more.
  5. Define deployment, operations, etc.
    Operations implements ITIL, we should interface with it as it matures.

We deliver.

Abstraction Layering – architect perspective (how)

Architecting abstraction layering is done to serve the data integration projects by empowering a few roles, these includes:

  • Project manager
    Work breakdown structure gets easier because one for any integration task have some nice metaphors.
  • Developer
    Gets assignments with a predictable scope.
  • Tester
    Can reuse tests because many atomic items has the same scope.

The architect gets a bag of expression for reuse in the modeling of all the ETL jobs and test templates. It gets possible to create templates for kinds of functionality used often or placeholders for functionality other systems depends on.

Abstraction Layering – project manager perspective (what)

The developer lead gets measurability and some nice metaphors from the abstraction layering, in our current project they are

  • Job
    Roughly equivalent to executable, e.g.: "Nightly job".
  • Agg
    Typically one for each of the different states a job goes through, e.g.: "Source extract"
  • Fun
    Specific function for an entity, e.g.: "Extract <<customer tables>> from source system X"
  • Task / Tsk
    Task is part of a function, it moves, changes or collects data. Data in the warehouse knows where it comes from by reference to a such task id, a sample task might be "Extract customer address from source system X".

The project manager must choose when these metaphors are appropriate, of course.

Abstraction Layering – developer perspective (what)

When assigned a task a developer can by the name of the delivery see how it fits into the wide picture on three dimensions

  • Job/Agg/Fun/Tsk
    Dictates the level along the low-level to high-level axis.
  • Context
    Horizontal layers in the architecture touched, for instance SourceDsa or DmsaDm.
  • Function
    Typically the 'T' in "ETL".

Most work repetitive by nature should have current templates controlled by the architect.

Abstraction Layering – test perspective (what)    

Too early to say, but it looks like a combination of the abstract layering, "data haven" and MR shall make test-driven development beyond simple testing possible. Looks like integration and performance testing shall come relatively cheap.

Fading out

OK, this grew pretty long, looks like I'll have to do more on parts of this later, with more concrete samples. Hope I find the time.

G.