Monitoring server

Server for monitoring IT services is crucial for effective management of IT infrastructure and to guarantee SLA. Two years ago we developed something like that and in the end it was realy usefull piece of software. ZABBIX is very interesting open-source project for network and service monitoring. For further information check out http://www.zabbix.com

Fedora Core 5 on USB hard disk

I have DELL notebook Latitude D610 with preinstaled company standard image. We have lot of restrictions and security constraints in this image. It's absolutly correct from security and company's point of view. But because I'm working as solution/system consultant I have to test a lot of systems, softwares and solutions to have a practical experiences with latest technologies. So I had two possibilities. To buy another hardware for testing or use a company notebook with another hard disk and another OS. Second posibility is chiper and more convinient for me. You can pretty easy demount internal HDD from DELL Latitude 610 but it's not so professional. Much better way is to use external USB hard disk and during the boot press F12 key and choose a boot device. Because we are suggesting virtualization tou our customers I decide to install linux and VMWare. I choosed linux distro Fedora Core 5 (FC5) as hosted OS. There are some hacks haw to install FC5 to external USB disk. First of all you have to choose different type of installation method from DVD/CD medium. You have to use option "linux expert" which is not sugested in instalation wizard. Only "linux expert" and "linux rescue" are loading USB modules so it's working with external USB disk. After common CD/DVD installation you have to reboot. But after reboot system is not booting propperly from external hard disk. You can see "kernel panic" on the screen. The reason is that generic instalation doesn't contain USB modul. There is work-arround. You have to boot from CD/DVD to rescue mode (linux rescue). Installed system is mounted to /mnt/sysimage/. You have to mount boot image by command
mount /dev/sda1 /mnt/sysimage/boot
set a chroot directory
chroot /mnt/sysimage
and create new initrd image with modules for USB storage
mkinitrd ––preload=ehci–hcd ––preload=usb–storage ––preload=scsi_mod ––preload=sd_mod /boot/initrd-2.6.15-1.2054_FC5.img 2.6.15-1.2054_FC5

This work-arround change booting image to new boot image with USB disk support and after reboot Linux OS should boot from external USB hard disk without any problem.

VMWare instalation to be continued ...

Business proceses and Workflow Software - Part II.

Continued from part I. ... We found (with my collegue Juraj) much more enterprise software suits for BPM. It looks that BPM is trendy and lot of vendors try to catch market share. You can find these vendors in Gartner Magic Quadrant: Tibco Software, Lombardi, Savvion, Pegasystems, Fuego, IBM, and others. After our "hot" discussion about the best tool to use for workflow application we decide to use OpenSource JBoss engine jBPM and develop other components in-house in SUN Java Studio Creator which looks like pretty interesting RAD (Rapid Application Development) tool for web based applications. The reason of this decision is that we have programmers background and it looks like best way for us. It's not a common way of course :) I will write later about our experiences.

DELL & JBoss Partnership

Do you known that DELL and JBoss have a partnership? See at http://www.jboss.com/partners/dell and http://www.dell.com/jboss

Marc Fleury's (JBoss Founder, Chairman and CEO) published at his blog a really funny and absolutly typical DELL story because this is the way how DELL is working. DELL is always waiting until particular technology is standards based, commoditized, with business perspective and without potential business, TCO, and ROI impact. Read more at Marc Fleury blog http://jboss.org/jbossBlog/blog/mfleury/2005/08/24/DELL_I_love_Systems.txt

Business proceses and Workflow Software

I believe to process-driven business to be successfull in long term perspective. The main key are right designed core and supported business proceses. When proceses exists you have to follow up them. There isn't big issue to design BPs but you need some tool for be sure and have a full control if everything is going on the right way. So you need some workflow software. The basic idea of workflow is not so complicated. From mathematical point of view it's just oriented graph. I have two main requirements for this software tool. Firstly, it must be good in worflow design (easy import already designed or refactored BPs). Secondly, system must be open for integration with other systems like mail and messaging systems, document management systems, and so on. I found two interesting product JBoss (http://www.jboss.com) nad WokflowGen (http://www.workflowgen.com). I want to spent some time to study and practicly check what benefits these solutions giving us. I personally prefer open source solutions with comercial support so JBoss have small advantage. I'll write full conclusion later.

DELL | EMC storage products

I have just passed all EMC eLearning tests which are necessary to get EMC sales certification. Since 15 May till 20 May I was in EMC New Hire Trainig in Cork, Ireland to get first overview of DELL | EMC storage products. I was really impressed what is possible to do with inteligent disk arrays. The true is that products are pretty expensive for small busines but on the other hand the functionality is incredible and SMB or enterprise customers who knows what is the value of information can benefit from this technology. I really like functions like snaps, clones, and synchronous/asynchronous mirrors. And what disk space you can use in those arrays? Depends on model. For price sensitive customers there are models DELL | EMC AX 150/AX 150i with up to 12 SATA drives which means 6TB of raw capacity if you use 500GB disk. Difference between AX 150 and AX 150i is connectivity. AX150 have fiber channel ports and AX150i have ethernet ports and works as iSCSI tagets. Enterprise model CX300 is a member of previous generation but it's still available and can handle up to 60 Fiber Channel (FC) disks. New geeneration models CX3-20, CX3-40, CX3-80 can mix up FC disks and Low Cost Fiber Channel (LC-FC) disks. CX3-20 support up to 140 disks, CX-40 up to 280 disks, and CX-80 up to 480 disks. Imagine that today you can buy 300GB FC and 500GB LC-FC disks. You can calculate usable disk space by your self. The main message is that with such products is server and storage consolidations significantly easier. Better days for administrators are comming.

Modern Software / Moderní software

UPDATE: This English translation was produced by ChatGPT on 2025-05-17 from the original blog post dated 2006-06-24. The original Czech version appears below the translation.

We often talk about a “working application,” but today’s applications—or more precisely, software systems—are no longer as simple as single-user DOS applications once were. Today, we expect software systems to handle a high number of transactions per second, support hundreds or ideally thousands of users, and be multi-tiered and integrated from multiple interconnected subsystems using the best integration tools. This makes them scalable and robust. You may have heard these terms before, but who can really make sense of it all? You practically need a team of experts to run such a system.

Who exactly do you need? A software architect, analyst, developer, operating system administrator, database administrator, and network administrator. Great—but now you’re facing monthly costs so high that you’ll need a really solid business plan to cover the overhead of all these people and still make a profit. But even if you’re lucky enough to have such a team, are they capable of quickly and reliably identifying every operational issue in the system?

Let me illustrate with a specific case that detecting a malfunction in a software system is not always straightforward. For years, we have been developing and operating an online airline ticket reservation system for a customer. The system is divided into several mutually communicating layers. The layer closest to the customer (let’s call it Layer 1) is a classic web system that uses object-oriented techniques (inheritance) to extend a simpler web layer (Layer 2). Layer 2 contains all the logic for the user interface but not the full visual appearance, text, or other customizations—that's all handled by Layer 1. This setup is ideal, especially since there are many customized websites built on the same foundation.

Both of these layers provide only the graphical user interface, while the reservation logic itself resides on the application server (Layer 3). Layer 2 communicates with Layer 3 using XML over TCP/IP. Layer 3 simulates the work of a live travel operator and attempts to find optimal flight connections in the AMADEUS international reservation system based on user requests. Layer 3 communicates with AMADEUS via its proprietary API over TCP/IP. It also has to store certain information and uses a database server for this, also connected via TCP/IP.

You’ll agree that this can no longer be considered a simple software system. From a hardware perspective, at least three servers are used: one web server for Layers 1 and 2, one application server for Layer 3, and one database server. All of them must be connected via a network. And this doesn’t even include the AMADEUS servers located on the other side of the globe, or additional servers used for load balancing, backups, automatic email notifications, DNS servers, and so on.

I’ve described this “relatively simple” system just to show how difficult it can be to diagnose a suddenly appearing issue. We had been developing and maintaining the software for about five years, while a third party was responsible for its operation. About a year ago, after a period of dissatisfaction, the customer decided to end the cooperation with the third-party provider and asked us to take over system operations as well.

Anyone who has ever taken over an existing information system knows that it’s no walk in the park. For us as developers, it was a bit easier because we understood the system architecture, but even so, some "improvements" had been made on the production servers during operations, and no one really knew much about them. Fortunately, we were able to arrange a gradual handover of the servers and document those “enhancements” with the former operator.

I won’t hide the fact that the server transition has been going on for nine months already, but since the system works, there was no rush. The transition involves preparing new hardware and reinstalling everything from scratch—while also upgrading hardware, the OS, and required libraries.

Now we’re getting to the core of the story. At one point, the web and application servers were still in the old provider’s data center, while the database server had already been successfully moved to our own. The system operated in this setup for several weeks without issues.

Then the client called, complaining that the web reservation system was occasionally not working. After further diagnostics, we found that sometimes the system worked fine, sometimes it was slow, and sometimes it didn’t work at all. Log files showed issues between the application server (Layer 3) and the database server. The application server logs had many errors about failed database connections.

I started reviewing what had changed in the hours and days leading up to the incident. The day before, we had deployed a new version of the application server, but the changes were minor and should not have affected the database connection. Still, I reverted to the older version just in case—but that didn’t help.

So I focused on the database. Its logs showed numerous failed connection attempts from the application server. After too many failures, the database blocked further access from that source. I then tried modifying some configuration parameters—increasing the number of allowed connections, increasing the client connection timeout, and so on. After about two hours of trying everything, there was still no improvement.

Then I remembered an email from our ISP a week earlier, warning of a 5-minute outage scheduled at 5:00 AM. I immediately checked my inbox and confirmed that the outage did occur that very day—and it matched exactly with the first connection errors in the application server logs.

We had acknowledged the email at the time, but our systems typically handle such short outages gracefully. Our central monitoring also reported no issues. But you’ve probably guessed it: since the application and database servers were in different data centers (with different ISPs), it turned out to be a network problem.

I launched a more aggressive ping test (ping -s 1000 -i 0.01) from the application server to the database server—and discovered about 10% packet loss. That explained everything. With traceroute and ping, I quickly located the issue: it was between our data center’s router and our ISP’s switch. In 99% of such cases, the culprit is mismatched duplex settings on the network interfaces.

Our interface was set to auto-negotiate and had settled on 10 Mbps half-duplex. A call to our ISP confirmed that their side was set to 10 Mbps full-duplex. I changed our interface to 10 Mbps full-duplex—and the packet loss disappeared. Instantly, the software system was working again.

I wiped the sweat off my forehead and reflected on the situation. The root cause was that the ISP had not saved the switch configuration we had agreed upon months earlier (auto-negotiation enabled). After the power outage, the switch reverted to the old settings.

I’m lucky to be from a generation of IT professionals with a broad skill set—able to handle all these steps on my own. I can’t imagine a similar problem being resolved efficiently by a team of separate OS admins, DBAs, application developers, and network engineers.

Long live simple systems! I felt like shouting, “WHO CAN POSSIBLY UNDERSTAND ALL THIS?” But we’ll simply have to accept it—and all of us in IT must continuously educate ourselves so we can quickly identify the true causes of problems.

The worst situation is when you don’t know what’s causing an issue and just say, “Well, I guess it’s just something between heaven and earth.”

Dell, as you may not know it / Dell, jak ho možná neznáte

English translation by Google Translate

My new employer will be Dell from May 15, 2006. Dell is known as a hardware vendor. Dell servers, desktops, laptops, PDAs, etc. are renowned for their stability and quality. Dell is less known as a solution provider. However, Dell has a special division - Dell Professional Services (DPS) - which provides services in areas such as large SANs (storage area network), HPC (high-performance computing), HA (High Available Computing), server and data consolidation, monitoring infrastructure, services in areas such as Microsoft Exchange, SAP, Oracle, Linux, etc. DPS is a global division, and therefore DELL has a local division in each BU (business unit), focused on the above comprehensive services. This group, formerly called "ASG - Advanced System Group", is now called "Solutions". I am currently entering the Solutions division as a Solution / System Consultant.

Update: After joining DELL, I found out that ASG / Solutions is a technical pre-sales department and not a DPS - Dell Professional Services. There was not even a DPS division in the Czech Republic at that time. Technical presales were not so new to me, as I also had to be able to sell our technical solutions in my company, but the concept of multinational presales was definitely different. I have to admit, I quite enjoyed it. However, after about two years, DPS was born in the Czech Republic and I was able to move from presales activities to the design and implementation of real projects for customers.

Original post in Czech language

Můj nový zaměstnavatel bude od 15.5.2006 firma Dell. Dell je znám jako dodavatel hardware. Svojí stabilitou a kvalitou jsou proslaveny Dell servery, desktopy, notebooky, PDA, apod. Již méně je Dell znám jako dodavatel řešení. Dell má však speciální divizi - Dell Professional Services (DPS) - která poskytuje služby v oblastech jako jsou velké sítě SAN (storage area network), HPC (high-performance computing), HA (High Available Computing), serverová a datová konsolidace, monitoring infrastruktury, služby v oblastech jako jsou Microsoft Exchange, SAP, Oracle, Linux apod. DPS je divize s celosvětovou působností, a proto má DELL v každé BU (busines unit) lokální divize, zaměřené na výše uvedené komplexní služby. Tato skupina, která se dříve jmenovala "ASG - Advanced System Group" se nyní jmenuje "Solutions". Právě do divize Solutions nastupuji jako Solution/System Consultant.

Update: Po nástupu do DELLu jsem zjistil, že ASG/Solutions je oddělení technického pre-salesu a nikoliv DPS - Dell Professional Services. Divize DPS v ČR v té době ani nebyla. Technický presales pro mne nebyl zase až tak něco nového, jelikož jsem ve své firmě musel rovněž umět prodat naše technická řešení, ale pojetí presalesu nadnárodní firmy bylo rozhodně jiné. Musím uznat, že mě to docela i bavilo. Nicméně asi po dvou letech se DPS v ČR zrodila a já mohl přejít od presales aktivit k designu a implementacím reálných projektů u zákazníků. 

Professional milestone / Profesní milník

ENGLISH TRANSLATION by Google Translate

Just today (April 8, 2006), actually yesterday, I received the final offer from one large multinational IT company (Dell) for the position of solution consultant. The job description sounds very interesting and the financial conditions in combination with other benefits are more than pleasant. Therefore, I decided to hang my own IT startup after five years and rest for a while (meaning with exaggeration) in the corporate sphere. 

We have done quite interesting and successful projects in an IT startup focused on internet and intranet solutions based on open source products. Recently, however, the payment morale of clients, permanent stress, the constant search for quality people to the team, and especially the inconsistent corporate vision and strategy, which we were unable to agree with the second co-owner of the company, made me take this step. I believe that with this step I will have more free time for my own interests and possibly also for open source projects, which I find interesting. Alternatively, I would like to focus on three projects, which I hope to report on this blog. This is an embedded OS project for routers www.wibsd.cz, a PHP framework project for web applications, and possibly a digital library project (software for archiving digital objects). 

In addition, I was approached by another very good friend who would like to start another completely different project. So we will see what happens to the time, which is definitely not a lot, but it is already clear that it will be necessary to set priorities because it is simply impossible to address everything. In addition to professional life, one also has a private life that cannot be forgotten.

Bloguju, blogujes, blogujeme ...

Již delší dobu čtu v různých časopisech a webech, že psát blogy je in. Ono se řekne psát blogy! Začít je celkem jednoduché, ale pravidelně o něčem psát, a ještě aby to mělo hlavu a patu, to už tak jednoduché nebude. A o čem, že bych to chtěl vůbec psát? Bude to, jak jinak, hlavně o IT, které je již dlouhá léta a dalo by se říct i desetiletí mým koníčkem i povoláním. Takže uvidíme jak to s tím blogem nakonec dopadne.

******** ENGLISH TRANSLATION *******

I have been reading for a long time in various magazines and websites that write blogs is in and trendy. It is said to write blogs! It is quite easy to start, but to write something on a regular basis, and even to have a head and heel, it will not be that simple. And what do I want to write about at all? It will, as it were, mainly IT, which has been for years, and I could say even a decade of my hobby and passion. So let's see how this blog eventually turns out.