Two weeks ago our DB2 team had a real stress test. Without any particular reason DB2 Production server of one of our clients began to freeze or crash. Monitoring system falsely believed that the database was alive since port 50000 was available.

Since nothing was changed in the software, we decided that it was hardware or data related issue. We moved database to a brand new server. However, after completing this step and running our test scripts we were able to reproduce a crash with similar symptoms. This was rather frustrating, however it was a clear indication that the problem was in fact on the software side and could be related either to corrupted data or its processing.

In parallel with our own investigation a Level 1 PMR was opened with IBM. We have requested 24*7. Three IBM teams were working in parallel on the issue.

We have been analyzing server workload, queries and logs for five days and nights. As a result we were able to isolate the conditions responsible for the server crash. We have shared the results of our findings with IBM support team and here is their response:

Response from IBM

By the way, you (PointSource) have independently arrived to the same conclusion as our team.
We have a fix. Here is the APAR text we will use:

Abstract:
An OLAP query with a wide row may cause a memory corruption and crashes.

Detail:
If an OLAP query has a row that is greater than 65535 bytes wide,
then it may cause a memory corruption and crash.
OLAP queries require the buffering of data in memory, but a logic
problem when computing the buffer size chose a size that was too
small to contain the wide row.  The later results in memory
corruption when the row was copied into the (too small) buffer
overwriting information beyond that buffer in memory.
Since this is a memory corruption, there is not always a
consistent problem signature to uniquely identify the case since
it depends on what information gets corrupted.
If a memory debug setting is used, like db2set DB2MEMDBG=PROTECT,
then the corruption can be caught at the time of the overwrite,
with the following stack traceback:

sqlri_buffer_add
sqlri_scalag
sqlriSectInvoke
sqlrr_process_fetch_request
sqlrr_open

As for work-around, the only option is to reduce the width of the row flowing into SELECT statement, possibly there’s a different way you can write this query that doesn’t have this wide a row.

Additional comments

The source of the problem was in combination of 2 factors:

a)  More than 65Kb of data returned in one row of the result set

b)  Using of “PARTITION BY” OLAP function in the query which was generated by our BI tool.

Each of the operation (CAST and PARTITION BY) is safe by itself. But together, and under certain conditions they lead to the problem.

DB2 logs

Another weird aspect of the problem is that DB2 logs were lying. We were able to find SQL statements that caused the crash in logs, but these statements were absolutely different, unrelated to some particular table or table space, and sometimes they were even system SQLs, referencing system catalogs.

Since it was a memory related issue DB2 logs were inconsistent and unreliable pointing to random tables and SQL statements. The statements that we had in our logs were not always the ones that really caused the problem.

The real statement causing the crash appeared in about 50% of all crash logs which eventually pointed us in the right direction.

Effected DB2 versions

We were able to reproduce problem on DB2 LUW v9.7 FP2, FP 4 and FP 5. The problem should be fixed in next Fix Pack.

P.S.

Until the fix is released by IBM I hope that this info will help those of you who found this page in your search results. We posted this blog in an attempt to help other DB2 Administrators who may have the same issue.

And I would like to thank all PointSource engineers who were involved in the problem investigation. I’m glad to work in a highly professional and committed team.

About the Author Maxim Moroz (Maxim.Moroz@pointsourcellc.com) is a Database Architect with PointSource with overall 15 years in software development and administration. He pays a lot of attention to the aspects that effect application performance.  He is IBM certified in both IBM DB2 and IBM Lotus Domino products.

http://www.startwire.com/

To learn more about all the buzz visit:

CNN:

http://management.fortune.cnn.com/2012/01/19/when-you-apply-for-a-job-and-hear-nothing/

Tech Crunch:

http://techcrunch.com/2012/01/13/hate-when-companies-dont-provide-feedback-on-job-applications-startwire-is-for-you/

Mashable:

http://mashable.com/2012/01/10/startwire/

Money Monday TV segment, WSVN in South Florida:

http://www.wsvn.com/features/articles/moneymondays/MI94263/

LifeHacker:

http://lifehacker.com/5876796/startwire-automatically-updates-you-on-the-status-of-your-job-applications

Miami Herald:

http://miamiherald.typepad.com/worklifebalancingact/2012/01/how-to-land-a-job-or-promotion-and-find-work-life-balance-in-2012.html

US News:

http://money.usnews.com/money/blogs/outside-voices-careers/2012/01/04/4-little-known-tools-for-a-2012-job-hunt

Given this problem, several solutions have come to market to provide more accurate, easy to develop front-end visualization mock-ups, which can be created easily on the fly, and give the client actual tactile feedback before a single line of code has been developed.  The obvious benefit here is that in a rapid timeframe, a “true-to-life” representation of an end product can be presented, discussed, and tweaked, thus saving much more time on the design and redesign of the actual product.  Out of these solutions, my coworkers and I decided to utilize a product called iRise for our visualization needs, as it presented all the necessary tools we desired.

iRise is a two-tool package:  Studio, which is the development platform we make use of to construct our mock-ups, and Reader, which we can distribute, and allows any clients or prospects to view the created iDoc files.  Studio comes with all the expected UI components, and the newer versions even come with pre-created element bundles for common-case situations, accelerating or enhancing your visualization projects.  The highest level view of the project is the screen flow, where you can put together the skeleton of your application in a flow chart.  Each of these screen flow elements becomes a page within the project, where you actually build  your user interface.

The tool has a built-in data layer, so that any of your elements can pass data either to each other, or even between screens, allowing for a much more representative mock-up than you could have previously developed.  Utilizing data entered in one element, you can control the reactions and allowable actions for other elements.  Finally, there is built-in annotation functionality, so that any necessary documentation can be added directly to the mock-up itself.

While we chose this tool due to our personal needs, there are many others available which may be better suited to your particular business cases.  However, the key point to note here is that no matter which tool your business decides on, the visualization industry has come a long way from its roots, and should be taken advantage of in order to clarify your communications, jump-start your process, and prevent unnecessary redesign late in the game.

About the author

Brian has over 5 years experience in the IT industry, over a year of which he has served as a certified scrum master. He brings a technical computer science background to the job, allowing for better understanding of both client and development concerns.  He has been with PointSource for over 3 years as a project manager, and helped lead their transition into the agile development methodology.

But no matter who wins a given lawsuit, or what technology gets pushed aside, one thing is apparent – we need to consider more than one option when thinking through our next mobile application platform. Other features which we need to consider are: ease of coding and packaging, tooling availability, ease of maintenance, unified code-base (if possible), skillset availability, and so on.

Up until a couple of months ago I was seriously considering using the Dojo/PhoneGap combination to create a single HTML5 code base deployable to BB OS, Android, Windows Metro and iPad based devices. However, as I was following Adobe’s never fading commitment to the AIR platform I realized that the choice of Flex is as strong as it has always been. Adobe has never stagnated and always stayed on top of the game. As a result it kept improving AIR, as well as, created sophisticated tooling for HTML5 development. But before we dove into the details, let’s go back few years.

It is natural that users working with websites or web-based applications require UI to be responsive and interactive. Up until few years ago, this was achieved by using a scripting language such as JavaScript, put on top of a static language – HTML. In other words, while HTML was providing the static layout and content of a page, JavaScript was used to make it live by adding additional run-time behavior to HTML. This was known as DHMTL (Dynamic HTML). However, the plethora of browsers and different implementations of similar features brought up significant headache to the web developers who tried to create cross-browser solutions. In order to mediate these challenges, several frameworks were born targeted at hiding the cross-browser implementation complexities. Today, creating an interactive HTML page means using one of those frameworks rather than raw HTML coding as before. Examples of such frameworks are jQuery, Dojo, and etc.

Generally speaking, there are many technologies which allow creating mobile applications, but when considering cross-platform compatibility the choices are narrowed down mostly to Flex and tooling like PhoneGap and Appcelerator’s Titanium which allow to code in HTML/JavaScript/CSS and then package for each platform natively. Both technologies allow cross platform developing and packaging. So, how do we choose which one to use? After all:

• Both can be packaged for all mobile platforms (The new AIR 3, released four weeks ago along with Flash 11, features “captive runtime” technology that help developers distribute AIR apps to places where Flash isn’t welcome–iOS devices or the forthcoming Metro interface for Windows 8.)
• Both support rich set of UI components including charts and graphs
• Both support non-UI features such as geolocation, web storage, and etc.

With no clear winner, arguments can be made in both directions, so let’s look at cons and pros of each platform from a birds eye view.

HTML5

Pros:
While there are many advantages to using HTML, I am really struggling finding those which will distinguish it from Flex. One of the main advantages of HTML based solutions is the fact that they require no additional runtime as opposed to Flex which can only run in a Flash Player or in AIR. However, this argument is valid only if the solution is being accessed from a browser. In case of natively installed mobile applications this argument is not applicable any more. The reason is because natively installed Flash/Flex applications run on Adobe AIR which can be installed seamlessly as part of the installation process.

Cons:
HTML5 is still in draft, but quite close to reach the Candidate Recommendation during 2012. Some sections are already relatively stable and there are implementations that are quite close to completion. But if we look at HTML5 support by different browsers (see the resources below), we will realize that we are long way from having truly cross-browser implementations. While some of these challenges can be mediated with frameworks such as Dojo, the mobile market adds cross device issues as well. In other words two Android devices from different vendors can use different methods in rendering certain UI elements. The differences in rendering might not be noticeable on the big picture, but surely something that should be considered.

Also, the other day I got to play around with the demos and samples from IBM’s Web 2.0 Mobile Showcase (deployed on WAS 7.0) using my Galaxy 10.1 tablet. One thing I noticed off the bat is that every page rendering ended up with an annoying flickering. On top of that I was not impressed with the overall performance of the UI heavy demos. These will definitely not go unnoticed for users working with your application on daily basis.

Flex

Cons:
No doubt that Flex has its own corks. Over the last few years we have seen all sorts of problems ranging from weird layout problems to the protocol specific one, where the Flash Player would offset the timestamps in date type fields. Other problems included\ lack of page specific navigation and total application crash when something goes wrong, but all these problems can be addressed with proper coding and testing.

Pros:
Today there are many flash-based applications and games exist which can be downloaded from iTunes. This became possible after Apple removed the ban on Flash applications back in Sep 2010.

While being a single vendor standard Flex is open and keeps progressing supported by a single vendor implementation. This fact should allow us to have identical UI across all devices. Other important advantages for our current products are: existing assets which include application infrastructure, reusable code, and, the most important of all – our experience. Over the last few years developing PULSE and iEHS we have come up with many problems being layout or protocol specific. But thanks to our teams we have overcame those challenges and moved on collecting an important experience along the way.

Considering the thoughts outlined above, we have decided to use Flex as our mobile platform for iEHS. The prototype is yet to be created, but I am quite excited about the opportunity to expand our experience and build a totally new layer of services which we can provide to our clients.

About the author

Aram brings to PointSource over 15 years of experience in the IT industry. He joined PointSource in 2007, after working closely with the PointSource team for over 6 years on varying projects as a consultant. Aram is an experienced software architect who has been involved in numerous complex solutions involving multiple technologies. He has authored and co-authored multiple publications in both IT and science related fields. Aram holds an MS degree in Theoretical Physics from Yerevan State University, Armenia, with a specialization in Computer Modeling in Physics.

  for their research and contribution to this blog.

Resources:

HTML5 and Mobile
http://www.slideshare.net/WorkLightInc/…

Tablet devices and enterprise apps
http://www.riagora.com/2011/10/…

A whitepaper comparing Flex, Silverlight and HTML5
http://www.scottlogic.co.uk/blog/colin/2011/05/…

HTML5 Support in browsers
http://caniuse.com/#

More and more organizations are learning how to use game dynamics to engage customers, employees and partners. I thought it would be helpful to share a few examples, highlighting the game mechanics and provide links to learn more about each example.

SalesForce/Engage | Leaderboard, Achievements, Leveling

Salesforce has taken gamification to another level with this addition to their popular CRM platform. With Engage, Salesforce users activities within the system are tied to various game mechanics and offers direct competition with other users within their organization. These capabilities simplify complex performance metrics, motivate the sales force and provide clear direction for working the pipeline.

Intuit/Mint | Achievements, Progress Bar

Can tracking finances, managing budgets and saving money be fun? Mint.com figured out a way to not only build an easy-to-use personal finance website, but to attract more than 6 million users since launching in 2007 – 90 percent whom say they have changed their financial habits as a result of using the service.  Mint pulls all your financial accounts into one place. It allows the user to set a budget and track goals. It provides a Financial Fitness Score that is based on core game mechanics associated with task completion, progression and achievements. By taking an ordinary exercise and creating a casual gaming experience, mint is creating an opportunity to drive new user acquisition in a creative way.

Starbucks/My Starbucks Rewards | Leveling, Rewards

How did Starbucks transform a pedestrian commodity into a high-end accessory? They understood from the beginning that they were selling a lifestyle. Starbucks was not just a place to get a cup of gourmet coffee, but a center for socializing and intellectual discussion.  In a never ending quest to stay connected to the customer and keep them coming back, Starbucks has incorporated game mechanics into their popular loyalty program. The multi-level reward program includes a progression tracker that keeps consumers incented to continually engage with the brand.

The Swedish National Society for Road Safety/The Speed Camera Lottery | Rewards

Our world is full of negative reinforcement and punishment. Games teach us that positive reinforcement works better for creating most kinds of behavior change and adherence. Internationally-renowned designer Kevin Richardson leveraged gamification when he conceived the Speed Camera Lottery.  This innovative system rewards safe drivers by entering them in a lottery funded by speeding motorists’ fines. Over a three-day trial, almost 25,000 cars were caught on camera. The experiment was able to reduce the average speed of cars travelling through a school zone from 32km/hr before the system’s installation to 25km/hr.

Nike/Nike+ | Achievements, Badges, Challenges, Rewards

An iPhone app that maps your runs, tracks your progress and gives you the motivation needed to go even further. Includes features such as hearing mid-run cheers every time your friends like or comment on your run status, or lets you outrun them in a game of Nike+ Tag. The gamification of a healthy life style, not to mention selling shoes, apparel and accessories – Nike+ is just one more reason to choose Nike over competitors.

In today’s fast paced environment of changing business needs and rapidly evolving technologies, I recommend that you partner with an experienced information technology service provider who specializes in process improvement, has hands on experience with the right technologies, a clear understanding of the business issues you face and the experience to lead you to higher levels of business agility.

About the author

Esther Mattick has over a decade of experience leading process driven optimization projects that enable organizations to achieve revenue improvement goals in a cost effective manner. She has working experience as a Quality Assurance Technician, Business Analyst, Project Manager, Operations Manager and Director of Technology. Esther has established a reputation for making complex technology problems understandable and manageable.

Inspirations

Gartner Press Release, April 12, 2011

Gamification Summit NYC

The underlying technology that enables the management of business process is often referred to as the Business Process Management Suite (BPMS).  While the primary goal of BPMS is to monitor processes and look for ways to improve processes, those with a mature BI component are able to put decision making explicitly into the processes. This empowers the business to move beyond historical (what happened) and in-progress (what’s happening now) perspectives to a predictive (what is likely to happen) mindset where insightful forecasts allow the organization to optimize recommendations and automate decisions where appropriate.

So what makes up a mature BI solution? Look for the following key features:

Scalable and Unhindered Data Access – While your initial BI implementation may only impact a handful of end users, once the solution has proven its value, more and more users within your organization will want access to it. The number of different reports as well as the number of data sources the BI solution will need to pull from will also expand. Mature BI solutions support growth in all of these areas.

Tip: Look for scalable BI solutions that can easily gather and consolidate real-time information from sources that are proprietary, antiquated, or obscure.

Operational Visualization – A picture paints a thousand words. When dealing with large amounts of data in real time it is critical to have interactive dashboards that let you see how your company is doing and then drill down to analyze the reasons “why.” These highly visual summaries typically include charts, gauges, scorecards and reports all of which can be customized to provide personalized views of the information need to monitor and manage the business. Visualization makes it easier to interpret data and identify vital patterns.

Tip: More sophisticated BI solutions include the ability to transform data into matrixes, scatter plots, histograms, or maps.

Ad Hoc Analysis – For the power users who needs to answer an urgent question, make an “on the fly” decision, or address a pending issue. Mature BI provides features that allow even non-technical users to quickly and easily build and generate their own custom queries and reports.

Tip: Make sure the Ad Hoc user interface is easy to learn and accessible from any Web browser.

Advanced Predictive Analytics – Anticipate critical business trends and events. Move beyond “What If Analysis” and help your business discover subtle patterns and associations that allow you to develop and deploy predictive models to optimize decision-making and mitigate risks.

Tip: A few of the mature BI solutions include features that allow for rapid and highly accurate forecasting and planning.

In today’s fast paced environment of changing business needs and rapidly evolving technologies, I recommend that you partner with an experienced information technology service provider who specializes in process improvement, has hands on experience with the right technologies, a clear understanding of the business issues you face and the experience to lead you to higher levels of BI maturity.

About the author

Esther Mattick has over a decade of experience leading process driven optimization projects that enable organizations to achieve revenue improvement goals in a cost effective manner. She has working experience as a Quality Assurance Technician, Business Analyst, Project Manager, Operations Manager and Director of Technology. Esther has established a reputation for making complex technology problems understandable and manageable.

Inspirations

Finding Your Level of BI and Performance Management Maturity: BI and Planning for Midsize Companies – Introducing IBM Cognos Express, Featuring Research from Gartner

Role of Business Intelligence in Process Improvement

Business intelligence for business users: Insight when and where you need it

What to Look for When Choosing Enterprise Analytics Software

Even if you have just started to work with RDBMS, you may have heard that using indexes is good. While there is a plethora of information on the Internet, my goal is to choose and consolidate some basic knowledge about indexes into a series of articles. In doing so I’m hoping to provide a good starting point for beginner DBAs as well as valuable information for advanced DBAs too. In these articles I will cover some of the most frequently asked questions about indexes.

I’ve always want to know more about the system I’m working with, to perform tests and find facts that are not covered by regular documentation, but doing so is time consuming – as I’m sure you know. In this article I will help you understand how indexes can provide an order of magnitude improvement in performance by demonstrating how to test and analyze the impact of indexes. All this will be seasoned with a small amount of humor to make the education pass by a bit easier.

Some of the topics that we will cover include:

Beginner level:

• How index works?
• How index helps?
• How many indexes do you need?
• Can indexes kill performance?
• Which columns should be indexed?
• Does the order of the columns in WHERE clause matter?
• Does the order of the columns in index definition matter?
• How can I measure performance improvement after index creation?
• How do I know if index is NOT used for query processing?
• I created an index, but it is not used by DB2. What is wrong?

Advanced level:

• Compound index: Should I create one index for multiple columns or multiple indexes for each column?
• What does “INCLUDE <columns>” clause for an index means and how it can help me?
• What types of indexes are appropriate for you?
• …

So, what an index is?

Before we start to dig in, let’s talk briefly why indexes are so helpful and how indexes work. The best approach to learn something is to understand why it was done and why it was done in that particular way.

For example, you have a table with employee records stored (in no particular order) in files on hard drive. The goal is to get the list of employees records with DEPARTMENT=”Support”. How will you do that?

Let’s think like guys who invented the index. DB2 (or you) knows information about location of each table record on the hard drive. You can retrieve all records from the hard drive, go through them individually, get the DEPARTMENT value, compare it to “Support”, and get list of the records you interested in. Problem solved. However, you are getting claims from the users that it takes too much time to get the results. How can you improve it?

What if you can store somewhere just a DEPARTMENT value and location of the record on the hard drive? Since this is a very small subset of the data, it will take less resources to scan the hard drive and locate the records you need quickly. You have just invented an index. Additionally, indexes can be presorted and compressed for even more performance improvements.

In the TABLE, rows are stored without any particular order.

In the INDEX you can see that DEPRATMENT column values are stored in alphanumeric order. It is very easy to get all the records that belong to “Support” department. You know that you need TABLE records with row indexes (RID) 1 and 5 only.

So, what is an index? Index is kind of a small system table, with two columns: column(s) value, which is called KEY, and pointer to physical location of the corresponding record. An important note here is that the real column value is not stored in index as plain text. It is compressed and hashed. A wisely created index will always improve query execution in hundreds if not thousand times over non-indexed records.

How index helps?

1. Index provides fast and efficient method to locate specific records in a table.  It reduces execution time of a query.
2. Index stores data in logical order instead of physical. Key values are stored in ascending or descending order. It eliminates need to SORT values during query processing.
3. Index improves concurrency of the transactions. Records in the table can be located faster, as a result acquired locks will be released faster.

How many indexes do you need?

It is absolutely up to you. Index definitely brings performance improvements, but there are two things to be aware of:

First of all, when you create an index, you “duplicate” a column value in another place. This means, when you create an index for each column you use the same amount of space as the original table, plus some space to store data for record pointer for each index. So, space used by all indexes can be more than the size of the original table.

Secondly, problems appear when you create or update records in a table. Each update means that the index should be updated as well. If you have 10 indexes – DB2 will update them first and only then will proceed with update of the modified record. It forces your hard disk to find 10 small index tables, read them, and write changes back before it updates one original table record. A busy hard disk is a performance enemy.

Can index kill your performance?

The most common answer is yes. But, is this always true? My answer is – not really. Let’s run some simple performance tests to verify my position

Performance test:

For this test I chose a table with 300,000 records.

I created 5 indexes for all columns except LOGTEXT (VARCHAR(1024)), plus one PRIMARY KEY index. So, I have 6 indexes in total.

Then I did 10 INSERT statements and measured average execution time. In my case it was 62 ms.

After that I dropped all indexes and repeated the INSERTs. Now that there were no indexes to update, in theory, I was supposed to see some improvement here, but in fact I got the same 62 ms. Improvement was less than 1 millisecond, I did not even notice it.

To update one record, DB2 should perform 6 index updates (in real life I believe DB2 smart enough to see if column value was changed or not before updating it). In worst case scenario, execution time for INSERT or UPDATE statement will be increased in 6 times. Still, improvement from using the index is hundreds and thousands times. We are talking about spending of $6 and getting reimbursement for $6,000.

You can say that 300,000 records are nothing, and I should do my test on 300,000,000 records. Ok, perhaps in this case I will see a difference. Can you imagine a query processing time over table with such amount of records without an index?

Negative aspects of the indexes:

• Indexes require additional space.

If you have a 100 Gb database and you will create indexes for all tables, for all columns, the size of the database will be increases at least twice in size.

• A full index must be retrieved from the hard drive before it can be used.
• Insert / Update / Delete operations require updating the index data.
• Each index adds an alternative access plan that the DB2 Optimizer must consider during calculations of the optimum access plan.

Even with all these negative factors, large amounts of indexes will not kill your performance. They may negatively impact storage space, however.

So, index can not kill your performance. But, it depends.

Primarily, it depends on what type of database you have. Is it a warehouse (almost read only) or OLAP (often updates) system? If UPDATE execution time is critical, and you count each microsecond – a lot of indexes can be bad. If you do not care about that – then you mostly like have a regular, non-OLAP system, and your primary goal is to quickly send response on a user query.

Secondary, it depends on your goal. If you want to kill application performance via indexes – I am sure you will find a way to do this.

While indexes can bring performance improvement to your system, some times it’s more wise not to add indexes Be wise…

Which columns should be indexed?

As we discussed, index contains value for some column, which is going to be used in some kind of comparison. Any column used in WHERE, ORDER BY or GROUP BY clause is a perfect candidate for an index. Columns that have a Foreign Key definition are recommended for indexing too.

What if I have few columns in WHERE clause? Should I create a separate index for each column or create one index for all columns? Sorry, I will answer on that question in my next article.

Does the order of the columns in WHERE clause matter?

Does the order of the columns in multicolumn index definition matter?

Yes, it does. I will discuss this topic in detail in my next DB2 Tips Blog.

How can I measure performance improvement after index creation?

DB2 has an excellent product called the EXPLAIN facility, which can be found in mostly all DB2 related products.

Look for  icon in IBM Optim (InfoSphere, Data Studio) products, or “Access plan” icon in DB2 Command Editor (DB2 Control Center component).

This facility visually describes how DB2 engine proceeding your query. All you need is a SQL query which you would like to analyze. Here is an example of output.

Red TBSCAN octagon means place for potential improvement. TBSCAN stands for “Table Scan”, which means DB2 is not using any indexes for this part of query, it retrieves whole table and scans each record to see if it matches your request. In the properties of octagon you will be able to find which predicates (columns used in WHERE clause) were used. Create index for them and the problem will be solved.

Yellow IXSCAN, RIDSCN octagons mean that index was used for this part of the query.

Each figure in the following diagram has a numeris value representing the system resources spent processing; they are measured in timerons. It is not a time unit of measure. It is hardware independent unit of measure, which describes how many steps DB2 will perform to complete your query. In our example RETURN value s 2091. But, 2 servers with different hardware will spend different amount of time to complete this same query, which has the same RETURN value.

To measure improvement you should run EXPLAIN facility before you create an index. Then create an index, update Statistics for table AND INDEX (which is not by default), and run EXPLAIN again. If you did all right you will see that TBSCAN octagon is gone, and you have a yellow index SCAN instead of it. RETURN value will be changed as well.

How do I know if index is NOT used for query processing?

Run this query (DB2 9.1 for LUW and higher):

SELECT * FROM SYSCAT.INDEXES WHERE LASTUSED = ’0001-01-01′ AND TABSCHEMA NOT LIKE ‘SYS%’
ORDER BY TABSCHEMA, TABNAME;

All indexes in the result set were never used. You may find a lot of interesting information in that view. LASTUSED column represents when last time index was accessed by any SELECT query. If index was never used, it has value ’0001-01-01′.

In the same way you can check statistics for some particular index to see when it was used last time.

I created an index, but it is not used by DB2. What is wrong?

There are 3 main reasons.

1. You did not update table and index statistics.

RUNSTATS ON TABLE YOUR_TABLE ON ALL COLUMNS AND DETAILED INDEXES ALL ALLOW WRITE ACCESS ;

COMMIT WORK;

2. Based on the table and index statistics that DB2 have, it can decide that it will be faster to scan the whole table instead of checking some index. For example, you perform query like SELECT * FROM … WHERE DEPARTMENT_ID=200;

If  99.99% of all records has DEPARTMENT_ID=200,  it may be easier to get whole table from HDD and get rid of these 0.01% of records.

3. You forgot to include WHERE clause in your query. No conditions for search, no indexes involved.

Next time.

Today I have covered enough basics to get you going. Next time I will cover these topics:

• Compound index: Should I create one index for multiple columns or multiple indexes for each column?
• Does the order of the columns in WHERE clause matter?
• Does the order of the columns in multicolumn index definition matter?

And as always will do performance investigation.

About the Author Maxim Moroz (Maxim.Moroz@pointsourcellc.com) is a Database Architect with PointSource with overall 15 years in software development and administration. He pays a lot of attention to the aspects that effect application performance.  He is IBM certified in both IBM DB2 and IBM Lotus Domino products.

In my previous Cloud blogs, I provided a definition of the Cloud from a business perspective, explained how businesses were already utilizing the Cloud, described which applications are best suited for the Cloud, and what application traits send up a red flag when considering the Cloud. Today, I want to talk about how to develop your Cloud Strategy.

Let me start off by saying that you should focus on how the Cloud can enhance your existing technology strategy to better meet strategic business needs.  You should not be developing a standalone Cloud strategy, instead, include Cloud based solutions in your existing technology strategy where appropriate.

So, how do you go about figuring out where Cloud based solutions are appropriate for your business? First, if you haven’t read it already please read my blog The Cloud. What applications are good candidates?.

Second, I recommend you follow Forrester Research’s advice on how to appropriately position cloud offerings as part of your existing technology strategies:

1. Integrate IaaS into your infrastructure strategy

At the infrastructure level, the business problem is to cost-effectively provision infrastructure to run your business. IaaS becomes the right answer when, for a given business scenario, it provides the most effective computing infrastructure based on financing, flexibility, reliability, security, and other factors important to that specific business scenario. Evaluate IaaS options alongside on-premise data centers, private cloud, hosted cloud, traditional outsourcing, and managed hosting.

2. Integrate PaaS into your application platform strategy

For application platforms, the objective is to provide effective tooling to allow your solution delivery teams to build and maintain a coherent portfolio of business solutions and, where appropriate, to do specialized development for individual applications within the portfolio. PaaS is right for your business when its combined solution development and runtime environment compares favorably against other application platform options on cost, flexibility, developer skills management, and solution life-cycle management either for your entire portfolio or for individual applications viewed as part of your overall portfolio. In other words, the absolute best platform for a single isolated app is not always the best platform when that app is considered within a coherent application platform strategy.

 3. Integrate SaaS into your business solution portfolio and road map

When buying or building a portfolio of solutions to run your business, the business problem is to achieve effective, competitive business operations. SaaS is the right option when the portfolio fit, cost, risk, functionality, operational robustness, and integratability of a (small or large) SaaS-based solution allows it to provide the best combination of business effectiveness and cohesive fit into your application road map and strategy. An application that provides the perfect fit for a given set of business needs may or may not fit well into your solution portfolio. If it doesn’t, there’s the risk it will create upfront and ongoing integration and operational hassles both for IT and for your business process.

And last but not least, I recommend that you carefully consider the timing of introducing Cloud based solutions. Critical infrastructure (think hardware purchase) and application life-cycle events (build/buy new capabilities, replacing end-of-life solutions) should prompt you to consider how Cloud based solutions might play a role in improving business agility and operational excellence.

About the author

Esther Mattick has over a decade of experience leading process driven optimization projects that enable organizations to achieve revenue improvement goals in a cost effective manner. She has working experience as a Quality Assurance Technician, Business Analyst, Project Manager, Operations Manager and Director of Technology. Esther has established a reputation for making complex technology problems understandable and manageable.

Inspirations

Cloud Computing from Wikipedia

IBM White Paper, Dispelling the vapor around cloud computing

Top 5 Cloud Applications for 2010

To Cloud or Not to Cloud in Financial Services

The Confusion of IaaS, PaaS, and SaaS

Get A Strong Focus For Your Approach To Cloud

Do You Need a Cloud Strategy? – Part1

In my previous blog I provided a definition of the Cloud from a business perspective. In addition, I explained how businesses were already utilizing the Cloud even though they may not have realized this fact. Today, I want to talk about which applications are best suited for the Cloud and what application traits send up a red flag when considering the Cloud.

I believe that the number one factor a business should consider when identifying which applications are candidates for the Cloud is competitive value.  If the software provides capabilities that your competition also has then it is a strong candidate for the Cloud. This type of very low competitive value software includes collaboration applications used for email, chat, blogging, conferencing and file sharing. I like to think of these types of applications as commodities. Many businesses have already moved to the Cloud for such services.

Commodity applications can be readily supplied by a number of Cloud vendors, commonly referred to as Software-as-a-Service (SaaS) vendors. These companies are experts in the capability and can provide the capability cheaper. Lower total cost of ownership is achieved through the economies of scale provided by shared computing resources. These resources include hardware, software and IT staff. When you go with the SaaS model of the Cloud (aka Public Cloud) maintenance, upgrade projects, support and bug fix activities are taken on by the Cloud vendor.

It is important to remember that some of the software your business uses should be consider your company’s intellectual property. These truly proprietary applications are the ones that help you stand out from the competition and achieve revenue improvement goals. This software has a very high competitive value and is what your IT department should be focused on developing and supporting.

While you may consider running proprietary applications in the Cloud with vendors who provide infrastructure (IaaS) or platforms (PaaS) as a service, there two key considerations; latency and security. Applications that demand sub-second response times are not great candidates for the Cloud because of the increased distance between the user and the application. Second, keep in mind that some applications require access to or integration with other applications to share data. When the data is large in volume or highly sensitive, integrating in the Cloud can be both costly and risky.

In my next Cloud blog, I will talk about how to develop a Cloud Strategy for your company.

About the author

Esther Mattick has over a decade of experience leading process driven optimization projects that enable organizations to achieve revenue improvement goals in a cost effective manner. She has working experience as a Quality Assurance Technician, Business Analyst, Project Manager, Operations Manager and Director of Technology. Esther has established a reputation for making complex technology problems understandable and manageable.

Inspirations

Cloud Computing from Wikipedia

Peak 10 Presentation of Cloud Services

IBM White Paper, Dispelling the vapor around cloud computing

Top 5 Cloud Applications for 2010

To Cloud or Not to Cloud in Financial Services

The Confusion of IaaS, PaaS, and SaaS

It is clear to me from a recent seminar I attended that most people are uncertain about how to define the Cloud and that the whole concept has become more confusing over time rather than clearer. This is not surprising given that “The Cloud” has become an overused marketing concept claiming anything that runs outside the firewall. Meanwhile, techies tend to use a very narrow definition describing the Cloud as virtual servers accessed via the Internet – which is now more commonly being called “Cloud Computing.”

There are many flavors of  “The Cloud” such “Public”, “Private”, “Hybrid”  and several new emerging definitions of “The Cloud” most of which I place in the “Cloud Computing” category. Cloud Computing allows you to own your hardware (if you choose) and to build and/or own the software that runs on the equipment.  There are many variations to accommodate this approach. I leave it to your trusted IT advisor to understand these differences and make the appropriate recommendations.

To keep it simple remember that “The Cloud” (no matter which version you use) is built on the foundations of virtualization and the idea that software and infrastructure can be centralized on distant servers and remotely managed.

During the seminar the speaker asked for a show of hands for those who believed that their company was using the Cloud, most hands were not raised. I found this astonishing. I have been consulting for more than a decade and I can assure you that every business I have walked into that has computers has someone using the Cloud. Let’s just think about this for a minute. Most businesses use the Cloud today for one or more of their collaboration applications. These include applications for email, chat, blogging, conferencing and collaborative file sharing. Web Serving is another area where organizations have adopted the Cloud for hosting web servers and providing management and analytic tools. Many businesses have moved back-ups to the Cloud as part of their disaster recovery strategy. There are many other areas including CRM, think Salesforce.com, that businesses have opted to utilize in the Cloud.

Suffice it to say your organization is already using “The Cloud” whether you know it or not. The real question is what applications are best suited for the Cloud and what application traits send up a red flag when considering the Cloud. This will be the topic of my next blog…stay tuned.

About the author

Esther Mattick has over a decade of experience leading process driven optimization projects that enable organizations to achieve revenue improvement goals in a cost effective manner. She has working experience as a Quality Assurance Technician, Business Analyst, Project Manager, Operations Manager and Director of Technology. Esther has established a reputation for making complex technology problems understandable and manageable.

Inspirations

Cloud Computing from Wikipedia

Peak 10 Presentation of Cloud Services

IBM White Paper, Dispelling the vapor around cloud computing

Top 5 Cloud Applications for 2010