Python debugger has been elusive

Sometimes, when we are faced with a problem, we look for possible solutions and we may quickly resolve them. Then that same solution can be replicated to solve many other future problems of a similar nature.

That is my story about Python programming language. 

About three years ago, I had this little, but strategic, problem that required a strategic solution. I just heard from a friend that Python is very good for automation. So, I set out to read and research about Python ... and being a programmer, I was able to start using Python immediately, while I continued to build my knowledge of the language. Today, Python has become our language of choice for little, but powerful automation projects. I stress "little" for good measure.

Today, too, many of our users have expressed full satisfaction with the number of little, but powerful automation projects that we have developed using Python. For the end-users, of course, they have no idea that we use Python. In any case, it is none of their business.

The more we use Python, the more it becomes apparent that we need a fully fledged IDE for development purposes. So far, we use IDLE and PythonWin for writing our Python code. These are very basic editors, if you know what I mean; but they are the native editors for Python, so things like indentation - a key requirement for Python code - are well taken care of.

Recently, I was faced with a major hurdle in one of our numerous Python projects (one using Python 2.7 and AcGIS 10.1 - arcPy to be specific). I had been requested to make a major revision in the application, which I did using IDLE and PythonWin. Now, there were a few bugs that I needed to deal with. Unfortunately - but as expected - I was not to see any debugging tools in IDLE or PythonWin. This took me far backwards.

Again, reading and researching (of course using Google) are the key to knowledge acquisition. The dividend is that just yesterday, I just realized that Python has a debugging module called pdb (for “Python DeBugger”). Note that this is a module available in standard Python installation. This (pdb) was my savior because I was able to unearth the offending code in a matter of minutes, yet I had wasted several days before without a debugger.

To use pdb, first import the module like this:
import pdb

Then at the offending line, add the following code:
pdb.set_trace() 

You will then be able to do several standard debugging tasks. 

Listen to this: "A day that passes without learning something new is a day well wasted". I couldn't agree more.

QGIS has come of age!

Surely, QGIS (previously Quantum GIS) has come of age!

That is the statement I can make without fear or favor. All this because of the new functionality that come with QGIS 2.4. Frankly, I cannot hide my excitement at these developments.

I will not enumerate the new exciting functionality, but suffice to say that one of my biggest - if not only - problem with QGIS has been the lack of a good tool for generating good presentation-quality maps. I am not sure whether Print Composer has only just improved in QGIS 2.4, but I am really thrilled either way.

I have always had reason to fall back to ArcGIS whenever a map output was required, but now, I can achieve exactly the same - if not better - with QGIS. Print Composer is so much fun to use now, and the amount of flexibility it offers is amazing.

Whereas in the past I have had issues with Print Composer in QGIS as pointed out above, it has always been a breeze to use most other QGIS functionality, and in fact most of the geo-processing and data management tools in QGIS have always been so user-friendly compared to ArcGIS. I would sometimes do geo-processing or data-processing in QGIS and only revert to ArcGIS at the point of generating maps.

Now, I have no more worries about buying many licenses for ArcGIS. Just a few licenses will suffice to take care of highly specialized functionality that may only be available in ArcGIS.

The other beauty of QGIS is the vast availability of Python plugins that are very easy to install and use. I think it is safe to say "no turning back" at this point. I will now consider migrating some of my automated Python scripts from ArcGIS to QGIS and preserve ArcGIS licenses for "Super Users" only.

It's all about the power of opensource.

I am happy. What more can I say?

My experience in the last two weeks

It has always been obvious to me that the best way to learn (or perfect) something is by doing. There was no better testimony to this assertion than my experience in the last two weeks when I was faced with a situation that required my close (and direct) supervision of a data center. Look at a data center as a room full of servers, networking/communication devices, ACs, etc. I was not directly doing the dirty work, but I was initially watching and listening a lot (and researching on the web) - and eventually advising and directing with authority. So much in just two weeks!

My experience in those two weeks was nothing to do with database management or programming, or even GIS - things which are more or less part of my life. This time round, it was about building servers (both physical and virtual using hyper-V), networking, MS Exchange, data backup and restoration from tape using Backup Exec but with corrupted catalog, router and switch configuration, and so on and so forth.

This experience was a result of some system mal-function. Bad as it was, I always tend to look at the good side of things - just like they say "every cloud has a silver lining". And I can now agree that one cannot be an army general without going to the battlefield. I tasted (not tested) the battlefield in the two weeks, but I emerged a great winner.

I am now a much better person than I was two weeks ago, and I hope to become better and better. And I also realized that running a data center which outsources some of its services is not rocket science, but requires determination, good planning skills, attention to detail, and fair knowledge of broad technologies among others. A lot (not all) of the work done in the data center (at least the one I am referring to) involves using computer programs written by other people. I argue that you cannot claim to be doing rocket science if you are simply using a program written by others - where your  job is sometimes just to click Next, Ok, Cancel, etc.

I also confirmed that data center technical staff who work in a very stable environment may not learn much on the job, unless they are very creative and innovative. This is obvious and applies to any technical field. To achieve simple IT tasks, even though they may sound very simple, you will only be able to prove your worth at doing them if indeed you do them practically. Unfortunately (sarcastically), if none of your systems fail, you will not have the opportunity to prove your worth, and yet you may remain so contented that things are okay; if so, you are the real loser. This sounds really sarcastic, but it is the bitter truth .

Theory matters, but not as much as practice. Even if you are a supervisor or manager, get down from the "ivory tower" and at least witness how the foot-soldier is configuring the router, or even merely backing up data on tape. If you don't, you may even never know what a backup tape looks like. If you hear the techies talking their language - such as LUN, SAN, DMZ, you may feel out of place until you realize that these complex terminologies represent very trivial and obvious things. Demilitarized Zone? What a name! Although I knew quite a lot about DMZ long before, it was only in theory; I only got to understand it better in the last two weeks due to curiosity and unavoidable circumstances.

Occasionally, no matter who you are, get to the building rooftop to check what exactly an ODU is, and make a mental trip from the ODU to the IDU then to the router, or from the ODU to the base station via a line of sight. You will be on your way to becoming a general.

English Grammar vs Computer Programming Language Syntax

I pay quite a bit of attention to the English grammar, perhaps partly because I am a computer programmer and I am so used to the strict rules governing computer language syntax - which is good for discipline, I think.

Years back, I was teaching Advanced Calculus to Mathematics and Statistics students of one University, and I also had the chance to "teach or knowledge-share" English grammar with Literature students of another University during that same period under a rather informal arrangement. I had a lot of lessons to learn from the two separate classes, which spelt something like the digital divide we all know of. One thing I have always believed is that "attention to detail" is crucial irrespective of one's technical area of specialization. This brings back memories of excessive - and sometimes reckless - (mis)use of commas in written English, either through omission or commission.

Misuse of commas is very common, especially in newspapers - which sometimes makes me wonder what the role of Editors is. Painfully, the more readers see these things in newspapers, the more they come to accept them as the correct way of doing things.

I will give some examples shortly. But before that, let me emphasize that computer programming syntax (or grammar) gives no room for mistakes, lest you end up with syntax errors. For example, look at the following simple ANSI SQL query:

• SELECT * FROM PATIENT;

If you make a simple mistake and write the following:

• SELECT * FROM PATIENT,

You will not be forgiven because it will be a straight syntax error, and the query won't execute.

In written English, things are rather on the loose. You can easily get away with such mistakes and business will seem to continue as usual. Take the following statement, for example:

• Makerere University, is very famous.

This is obviously very wrong construction and abuse of the use of comma. There should be no comma in the above statement at all. Period! Yet you find such English construction (and lots of it!) in newspapers and written reports every other day.

Now, look at the following:

• Makerere University, located in Kampala, is very famous.

This is correct English construction. The phrase "located in Kampala" is considered passive and has to be delimited by commas as above. I could give several examples, but just one more would suffice.

• The president of Uganda, Yoweri Museveni, traveled last week. (correct English construction).

• The president of Uganda, traveled last week. (wrong English construction - abuse of comma - simply remove the comma).

I have seen this interesting example which should be able to further drive the point home. Here, I quote from a book which talks about the misuse of commas by comparing the following two phrases:

• Eats, shoots and leaves

and

• Eats shoots and leaves  

This is total confusion if you don't pay attention to the presence or absence of the comma. The first case could mean something like:

• Richard Orama eats, then shoots pool, then leaves the place (note that shoot is a verb here).

while the second one could mean:

• Richard Orama eats shoots, then leaves the place (note that shoot is a noun here). OR, it could mean Richard Orama eats shoots and leaves (i.e. he eats both of those things, both of which are nouns here)
  

Also note that in the first example, we do not know what exactly Richard Orama eats, but in the second one we know that he eats something called "shoots".

I hope we can all see the confusion. One should be extremely careful when deciding whether to add or omit a comma. This decision should be driven by what you mean to convey, but not just for the sake of it.

Just before I pen off, another common example is here:

• Richard Orama said, he is about to complete the project. (Wrong English construction - no comma please - this sounds more or less like a direct local translation).

Generally, as well, a comma can be used to avoid confusion. You will be shocked at how much your written statement will be (mis)interpreted just because you misused a comma. For example, do the following two sentences mean the same thing?

• Inside the house was a perfect match of civilization. (means that there was something available inside the house, and that thing was a perfect match of civilization).

• Inside, the house was a perfect match of civilization. (means that the inside of the house was a perfect match of civilization).

By the way, the semicolon and full colon are equally abused in written English. That will be the subject for another day.

Finally, as a data manager in the HIV/AIDS sector, one might wonder why all this fuss about English grammar. As long as you do some professional work, you will be required to write some kind of report at some point in time. It is important to be equipped with the knowledge of good English grammar .

Too much data, too little exploits

If I had the resources, backing and opportunity, I would advocate for a new breed of specialist data managers (in addition to existing data managers) whose role is simply to extract data and generate reports ... call them Reporting Specialists, BI Specialists, Report Generators, Data Extractors, or whatever you may like - but title is a trivia, so let's move on.

Through the times, I have learnt that many organizations own so much data which sits in some electronic database, but never gets the chance to see the light of the day. This really beats the purpose, and until we harness that data, we would be allocating resources to doing something that does not ultimately benefit the organization. To be specific, what I mean is that we would be spending a lot of money managing a lot of data for no ultimate use. Think about the resources spent on managing data - from human resources to software and hardware infrastructure ... the list is long. And I can assure you that this is the reality in many environments.

In my view, an average sized organization would do much better with a couple of Reporting Specialists whose role is purely writing SQL queries to extract data and format the output in some reporting format that can be used for research, M&E, management (strategic, tactical or operational) or other purposes. I can nearly guarantee that the Return On Investment (ROI) for spending on Reporting Specialists would be significant. We might all be amazed at how much positive impact such a move will have on the worrying HIV/AIDS rates in some countries.

Most organizations use electronic systems that were acquired off-the-shelf; a few build their systems in-house. Either way, the need for the Reporting Specialist is growing by the day. Without a paradigm shift, the concept of data-information-knowledge is grossly undermined because data will remain data, yet in the ideal world, data is meant to be transformed into information and ultimately knowledge. This is the real essence of data.

This is my take, even though insignificant as it may sound.

How can MS Access use a MySQL database as the backend?

This question arose while I was presenting a beginner's training in MySQL. It is very simple to accomplish this, but the answer is not one word, or one sentence. I will try to summarize below.

The technology used is ODBC (Open Database Connectivity)  - read about this for better understanding.

Assuming that you already have MS Access and MySQL installed on your computer, then do the following:

1. Install the MySQL ODBC driver - straightforward installation
2. Create the database with tables in MySQL - the normal way databases are created
3. In MS Access, setup ODBC driver to communicate with the MySQL database - simply using External data / ODBC database
4. Create your user interface (forms) in MS Access - as you always do! Only that this time the tables will be in a linked MySQL database.

In 4 simple steps, you are home and dry. With this setup, you benefit from the robust and stable engine of MySQL database management system, while you also retain the simplicity of forms design in Access.

Multi-user access to your database on a network will no longer suffer from the drawbacks of an Access database.

Remember also that this principle applies to any ODBC-compliant database management system (DBMS) - which most DBMSs are. In fact, ODBC drivers exist for most DBMSs, including Oracle, MySQL, Microsoft SQL Server, PostgreSQL, etc.

If you are interested in further details, read, read and read ... OR simply contact me. As easy as 123.

Beginner's guide to MySQL

A saying goes thus: "a journey of a thousand miles begins with a single step".

Likewise the journey to MySQL land may start right here, if you haven't already started. You will decide whether to reach your destination or not, but that is entirely up to you - largely driven by your interests and priorities.

In this post, I flag off that journey, but out of courtesy, you might want to share with us the progress that you have made in due course.

Recently, I made a presentation to the Data Managers Forum, and I thought that it might be worthwhile sharing. Please find the relevant document here.

OpenMRS - any challenges?

If you use OpenMRS and have faced some challenges, please post them here (using Add a comment). Our members should be able to answer a number of the questions raised. I will also consider inviting the OpenMRS developers to join in the discussion as necessary.

Demystifying GIS

In this post, I would like us to discuss and demystify GIS - Geographic Information Systems and at the end of it all, come up with ways in which we can use GIS to enhance research in HIV/AIDS and related fields.

GIS is an information system, and just like any other information system, it has components of hardware, software, people, etc.

The major difference between GIS and other conventional information systems is that GIS works with spatial (or geographical data) such as location of residence of a patient. Drilling down to databases, a GIS database is much like the conventional database, except that it has spatial attributes (fields) in addition to the ordinary attributes.

Several GIS software are available, the main commercial one being ArcGIS (developed by ESRI). There are some good opensource solutions such as Quantum GIS. You nay google, download and try it out. There are also Map Window, GRASS, etc.

A common misconception is that GIS is all about maps. Nothing could be farthest from the truth. GIS lets you visualize, question, analyze, interpret, and understand data to reveal otherwise hidden relationships, patterns, and trends in more ways than just maps. GIS is a diverse, versatile field with lots of background components of mathematics, statistics, data management, and geography. And if you like these technical subjects, then GIS is surely the place to be.

What is the difference between SQL and SQL Server?

Simple answer:

• SQL (Structured Query Language) is the language of relational databases, and can be used by any relational database management system.
• SQL Server is a database management system.

Therefore, you can use SQL for any database management system such as SQL Server, MySQL, Oracle, etc.

There is standard SQL (also called ANSI SQL), which applies to any database management system. However, database management systems have their database-specific SQL in addition to the standard SQL. This introduces some complexity in using SQLs especially if you use many database management systems in your day to day work. It is like speaking many languages at the same time. You are bound to make a few language errors.

What is it about Datafax?

This is simply a question ... some answers are expected from those who know better. And some debate may follow as well.

A glimpse at the statistical side of things

Amidst the data managers is a diversity of professionals, including statisticians. In this post, I implore any of the members to give some insights into statistical analysis of HIV/AIDS data so that other members may benefit.

Designing reports based on an electronic database

I would like to create a debate on the above topic.

The scenario at hand is that you have an electronic database (maybe Electronic Medical Record System, such as OpenMRS). Your users want online, real-time reports available to them. What possibilities do you have at hand?

I am not providing any answers right now, but with a vast number of options available, the debate can easily begin now.

Just to hint, a commercial reporting solution which I use is SQL Server Reporting Services (SSRS); the opensource counterparts that I use are JasperReports (with iReport designer) and Business Intelligence and Reporting Tool (BIRT).

Please have your say  ...

Launching the Blog for Data Managers Forum

Today, 15th March 2014, I am privileged to create and launch the blog for the Data Managers Forum, a new innovation aimed at providing data managers with a common platform for sharing ideas, both technical and non-technical in nature.

This blog comes as a result of a long deliberate effort to bring together data managers in the HIV/AIDS and related fields in Uganda. Since today we are in a global village, contributions to this blog will be open to whoever may wish to contribute across the globe.

This blog is not an official instrument of any institution, and it is not directly affiliated to any organization, although the initiators are data managers from the Infectious Diseases Institute, Makerere University, Kampala.

It is hoped that with time, interested organizations will be able to provide support to the Data Managers Forum (and by extension the Data Managers) in whatever way may be possible. In the interim, I will be personally moderating this blog to ensure compliance with acceptable standards - technical, ethical or otherwise.

I encourage members to post on a wide variety of subjects, typically (but without limitation) data management (collection, storage, analysis, quality assurance, reporting, etc), statistical analysis, programming, software development, database management systems, IT-wide issues, GIS, etc.

Remember "all work and no play makes Jack a dull boy", so some fun posts are warmly welcome as well.

Once inside the log, please post new entries (if you are a forum member) and/or make comments on posts (whoever you may be).

Richard Orama,
Founder of the Blog for Data Managers Forum.