Javamelody performance & usage statistics

One of the hidden gems in the open source world is a project called Javamelody.

I've been using this since late 2009 to help refactor/modify design and code based on usage-based findings. It is not a profiler, not a click-n-fix, not a quickly-fix-your-problems tool. It is a tool to get you the information, over time, that you need to make Strategic decisions about design/code.

http://code.google.com/p/javamelody/

It gets all tiers of statistics within a single application -> the application's UI calls, business (ejb/facade/spring) calls, and sql calls.

Recently I finally submitted a patch for GWT-RPC detailed statistics I've been using for a while to help, again from a strategic point of view, refine some products.

Enjoy!

Web UI upgradability

One of the areas that has been an issue over time is taking an application, say deployed to jboss 3.0 or 3.2.3, and try to upgrade it to jboss 4.0.5. Or tomcat 4 to tomcat 5. Or any upgrades at all.

Real-world experience with struts (1.0/1.1)/JSP sites have several challenges in upgrading. Whether they are container based or implementation base, I have never had success with 'easy' upgrades.

JSF appears to have similar issues. I myself kept running into performance issues everytime I've attempted a JSF implementation, so relying on this post to confirm similar issues: http://jsfunit.blogspot.com/2010/12/jsf-on-jboss-as6-final.html

Now, onto one known savior - GWT is upgrade compatible. I have successfully upgraded 1.3 to 1.5, 1.5 to 2.0, 2.0 to 2.1, and 1.3 to 2.0 (I haven't tried direct to 2.1). The only upgrade issues were 1.3 to higher versions dealing with RPC changes.

Testing GWT-RPC, and why to be careful about jumping to RequestFactory

Just a copy of what I posted on StackOverflow: http://stackoverflow.com/questions/4119867/when-should-i-use-requestfactory-vs-gwt-rpc/4714437#4714437


The only caveat I would put in is that RequestFactory uses the binary data transport (deRPC maybe?) and not the normal GWT-RPC.

This only matters if you are doing heavy testing with SyncProxy, Jmeter, Fiddler, or any similar tool that can read/evaluate the contents of the HTTP request/response (like GWT-RPC), but would be more challenging with deRPC or RequestFactory.

openEJB unit testing for jboss deploys

Some notes using mavenized /src/main/resources/META-INF/openejb-jar.xml:

<openejb-jar>
<!-- make backward compatible with jboss style deployments. For EAR deploys prefix the .format = EARname/{deploymentId} -->
<properties>
openejb.deploymentId.format = {ejbName}
openejb.jndiname.format = {deploymentId}/{interfaceType.annotationNameLC}
</properties>
</openejb-jar>

Eclipse JPA tooling, Hibernate (jboss) tooling

Working on ways to improve the tooling/work environment when in a JPA project.

In the past, pretty much hand-code everything and rely on maven/unit-tests to catch errors.

Quicknote experiences:

* To get JPA Tooling working, need to map the jdbc driver manually/directly to the filesystem jar location through the Eclipse->DataManagement features.

* More on JPA tooling, particularly with maven layout, here: http://www.eclipse.org/forums/index.php?t=msg&goto=508143

* To get Hibernate Tooling working, need to add the jdbc driver to the classpath, EVEN IF you are using Database Connection:JPA project configured option (i.e. see above direct jar filesystem mapping does not carry over to Hibernate Tooling).

* In the persistence.xml, to avoid dealing with a lot of issues, remove JTA requirements. This works for me as the Entity class/domain are in a project seperate from the Session Bean (the Entity Managers), so the Entity class/domain has a non-JTA persistence.xml, while the Session Bean (entity manager) project has a JTA persistence.xml. I hate inconsistencies, but only way this seems to work.


Gains:

* In JPA tooling, immediately checked the model to the database structure, and identified a couple of case-sensitivity issues between the field name and the column name that were easy to fix.
* In Hibernate tooling, can test-run jpa-ql queries to see if they work as expected, timing, and review results. Can also look at the Dynamic SQL Preview to see the actual sql used for future index optimizations.

(CI) Building Eclipse PDE plugins from Maven

...is a pain in the arse.

After evaluating maven-pde-plugin, which one would think would make it easy, turns out not so much.

I've swapped over to using Tycho (because it appears to better support multiple build options, like update sites and RCP apps directly instead of just plugins and features), but that isn't proving trivial even in the most basic sense still.

But, using Tycho 0.9.0 from the ibiblio org.sonatype.tycho groupid (not to be confused with org.codehaus.tycho...or several other groupId's I've run into) you still have issues:

Errors like: "Cannot find lifecycle mapping for packing: 'eclipse-plugin' come up a lot. Looking at the off-chance there is a dependency issue, you are required to use an unstable release version of Maven 3 (as of 7/26/2010 at any rate). Using maven 3.0-beta-1 you now get "Unknown packaging: eclipse-plugin"...so not much help there either.

Searching for help on either of these issues get posts like 'fixed in Tycho 0.5.0', or 'you need to modify how you build from source'...which if you get the binary from a public maven repository one would hope would work as expected (per why most people want to use maven so you DONT run into these issues).

Other people mention 'update m2eclipse'...except I'm running this from the command line for the purpose of eventually moving to Hudson/Continuous Integration. Maybe I mis-understand the purpose of this maven plugin and it must be used in eclipse with m2e?

Please help if you read this!

EDIT: reason for chasing down why I want to automate Eclipse PDE builds is

1) I have an RCP app I would like to migrate over (from Eclipse 3.0 unfortunately)

2) primary reason was to pre-load company JDBC drivers for use in Eclipse (http://www.eclipse.org/forums/index.php?t=msg&goto=549384)

ANSWER: do not assume the 'convention':


WRONG plugin artifactId: maven-tycho-plugin

CORRECT plugin artifactId: tycho-maven-plugin

Web Browser plugins, how I loathe thee, let me count the ways.....

I have had a passionate dislike for web browser plugins. Yes, they add new exciting features...that you may or may not be able to control, or have a predictable behavior across the world-wide-web.

Take for example two very common plugins that I usually have to deal with for reporting, document management/archive, etc.

PDF plugins (Adobe)

TIFF plugins (variety)

Adobe PDF plugins -

• Versions/upgrades regularly, users have to regular update 'the site', even though it's not the site, it is the plugin asking for upgrades.
• To embed, not embed, dealing with pop-ups allowed.
• And...here is a good one....the web-embed adobe plugin making *multiple* http requests for the same content, and if your logging didn't account for that - multiple logs (see http 206, byte serving/byte range requests).

TIFF plugins -

• Variety of plugins with different options/features/control (and even something 'simple' like if the plugin allows multiple page viewing....apparently not standard?!)
• You, your client/customer, or someone, has Outlook/Office installed and it has an update, a critical update, a security update, whatever -- and reverts to using the MS Tiff viewer by default despite your best effort to use a different TIFF plugin.
• TIFF encoding/compression formats (i.e. g3/fax compression that has an X-Y ratio difference, that some plugins understand and show 'correctly', and others that show without the appropriate ratio and have 'crunched' images).
• And the occasional TIFF that has a byte that isn't understood by Plugin XYZ, or other plugin, but yes on this plugin....search for it, they happen.

Then you add in other plugins like flash/shockwave, java applets, activex/silverlight, codec/encoding video players (whether to use quicktime, realplayer, windows media player, divx, ......), and developers just can not wait until HTML 5 becomes a real-world/real-usage deal.

BigDecimal v Float/float or Double/double for java transport

As I have posted previously, quite often I get involved in some type of financial portion of a solution, or the entirety of the solution is financial.

In java, BigDecimal is where you go for computational accuracy -- but what about if you just need to transport the data?

So, I reviewed information in the Sun/Oracle JDK site, and if you go search and read it, it isn't overly definitive (from a 'do I want to or not use') on float/doubles.

After going through many other posts, mailing list searches, and reviews, I broke down and posted a question here:

http://forums.sun.com/thread.jspa?messageID=10977271&#10977271

I also started doing some manual tests myself, and finally got the 'answer' I was looking for:

float: 9 'locations'

double: 15 'locations'

What are locations? My testing, I found that float can accurately store and retrieve 6 numbers before the decimal, and 3 after....or 3 before/6 after, or any variation of that theme. Similar for double - 9 before/6 after, and other variations.

Needless to say, that's why it is vague as it matters what scale you are storing after the decimal as to how much you can store before the decimal.

So, unless you can get a definitive max value and precision rule for a financial application, you might want to stick with the heavyweight of BigDecimal just to be sure.

----

Edit: I forgot to post *why* I was even looking at this!!

We were having some memory issues with an outsourced application (that lacked pagination), that had a DTO with 12 monetary value field...12 BigDecimals per DTO. The List sizes ranged from 300->2000->40k. The 40k (most extreme) was taking up 45MB of memory! Changing the BigDecimal to float primitive for the 12 fields dropped the same List size down to 15MB (1/3!!!!!).

However, the accuracy needed for this application was not satisfied by float, so although I'm evaluating Double I may opt to play it safe and keep accuracy as more important than saving memory (and, instead, actually paginate the results!).

Embedded DB - Sort Stability, Pagination

We use application-level pagination. I wont go into the reasons, but several of them are business reasons.

What is application-level pagination? Someone wants to view 50000 records through a web screen (just stay with me...business reasons).

• Make the query, default/starting sorting order.
• Cache results locally on the application layer the current set (in our case, cache into a hypersonic, h2, derby database that writes to file as too much to fit in memory).
• Return first results back to the web screen (say 50 records per page).

--person goes to 'next page', get next 50 records from local db result set.

--person re-sorts the existing resultset, re-sort from local db result set (instead of re-querying the origin db), return first .

Problems we ran into:

Certain embedded databases we found to not work out well for this challenge. Hypersonic and H2 both didn't see to handle (at least with default settings) the multi-user/asychronous(web/ajax) request nature of the sorts and were causing the result sets to not be accurate when 'pushed too hard' (a user requests a sort, then changes their mind in the middle of a sort and changes the sort again).

Derby however did seem to resolve this issue for us. Yes, there are different ways to handle pagination, however need to solve the business request of how the behaviour was expected to act.

If someone has some similar experiences with application-level caching of large result sets, re-orders, pagination, please share!

least-invasive development improvement

LIDI - Least Invasive Development Improvement (team-oriented)

Attempting to coin a term for what I have been attempting to do for the last 8 years in maturing a very small development team that supports many projects.

Definitions
Small development team defined as under 10 people, including UI, Server, DB, and internal dev QA.

Many projects defined as 10-25 active, supported solutions, with >50% of them being unique solutions, while the rest may be re-tooled/variations of existing solutions.

Small to medium projects defined (from a LOC standpoint) of between 10k and 500k. Most are web-based, but some are thick client. Most are 3-tier/n-tier, some are 2-tier thickclient-DB type solutions.

Key Words
*logging
*unit testing
*performance testing and review
*scalability testing and review
*security testing and review
*configuration management
*runtime management
*runtime dependency checking/management (i.e. notification of issues)
****Business problem solved
****Expectations met

Prefix
I put the last two, business problem solved and expectations met with many asterisks because, like many developers have experienced, doing all the performance/scalability/security testing in the world won't help you if you have to recode/redesign it again and then have to re-do all the performance/scalability/security testing and review again.

Discussion
When working with a small development team that is already fighting with project priority conflicts, short deadlines, short requirements, and constant support and change-requests, the last thing on your or their mind is ADDING more work.

The above term, least-invasive, is on purpose -- there is no free lunch, there is no silver bullet. There will be compromises, but if you maintain a goal of trying to make it as least invasive as possible, and able to show and provide reasons and results that are touchable/matter, you will mature and progress!

Experiences

Step 1: Baseline

I know your first thought - I don't have time to come up with baseline metrics, we are already going nuts! Guess what, I'm *not* talking about baseline metrics! I am talking about getting the development **process* repeatable and stable -- that is your base for everything you do.

*Baseline: Convention

Yes, I borrowed this term from the Maven team. Make sure all your projects follow a similar folder layout, for example all java code is in /src/main/java, all html/jsp is in /src/main/webapp, etc. Get the team to the point where someone can checkout a project they never touched before and be able to know where to go/what to do.

*Baseline: Independent builder

Either find a person who will always be an independent builder, or setup some type of continuous integration system. Having someone/something ELSE do builds than the developers will greatly stabalize the process and document/flush out any outstanding issues you have in the build process. This is a deadly experience I learned from my VB6, and now that I'm in java I've choosen http://hudson.net as an independent build tool, while continuum/cruisecontrol are other alternatives.

*Baseline: Build system/dependency management/versioning

I'm sure you just ran into a snag -- with an independent builder, you are learning there are different ways people are building, or worse, they are relying entirely on an IDE for the builds. Moving to ANT or Maven2 build system in java, in my case Maven2, helped to ensure that the builds are *consistent* and any gotcha's can actually get caught EARLIER than later. Let me say that again with an example - "This maven2 project is not building on my desktop, what a piece of crap." actually translates to "the project doesn't build on JDK1.4/JDK5/Windows/Linux/needs a library I forgot to add, lets fix it now while we're actively on the project instead of when we check it out 6 months later to fix a different issue.".

Maven2 also helps with the dependency management problem, and the versioning problem. If you are always renaming your jars to be mylibrary.jar to include in your application, and you aren't sure which version that library is after-the-fact and trying to identify an issue, you know the problem.

*Baseline: Promotion process

This will be the most difficult baseline to adjust - a promotion process. What I mean by this is, based on my experience that seems to be working, you develop and deploy to a DEV environment. Work out the kinks as you know it. If you are lucky enough to have an internal QA, have them review it on DEV. Then, when things look o.k., *promote* to a STAGING environment (including different DB, server, everything). NEVER make custom tweaks on Staging, instead always modify your promotion, or migration, scripts/process as those migration scripts/process is exactly what you are also testing that will be used when you promote to Live. On Staging, you do UAT/Customer Acceptance, have them push it back if needed, make fixes on DEV, them promote back up to Staging for another review. THEN promote to live.

Step 2: Improve ability to identify and fix basic stuff
What I mean by this, is let the developers use the tools they are already comfortable with. Unit Testing. Diagnostic Logging (or normal logging if you aren't familiar with the different logging types).

Unit Tests: Junit is a great. Nunit exists for the other side. Having a way to test the code is doing what you want to, AND BE ABLE TO REPEATABLY AND AUTOMATICALLY run those tests is the goal. This is not integration testing, just basic module/unit testing that the code is behaving as expected to whatever business expectations can be resolved in the code.

Diagnostic Logging: Making sure your code can log somewhere, that you can retrieve, and provide useful information to make a correction, is this goal. "It broke!" well, you need to know what caused it to break, and the *quicker* you can do that, the more time you'll have for other things. Rather than re-testing manually with system outs, get your logging taken care of. This will not fix all your issues, but if you can get the easy 80% out of the way, that's huge. My experience we are still having some challenges, as there are some custom ways that are already in place, and people have a hard time breaking out of the sysout approach. I think I'm satisfied with using SLF4J, then just letting the Log4j implementation for logging and controlling the log verbosity (and the formatting of the logs....nothing worse than custom logging that has many different outputs, get it consistent!).

Step 3: Your walking, your walking, lets try jogging.
By this point, you should be o.k. now, and taking care of business. Now should be able to look at some more technical things to improve the development process.

Codability: This is where the static code analysis tools come in, that are, again, not that invasive to use. Whether you do it from a report generation standpoint, or integrated into the IDE, tools like PMD, Checkstyle, FindBugs have the potential to ferret out potentially poor code. This is no replacement for peer review in any fashion at all, this is just a convenient way to identify common issues (note: these tools are not stone-cold rules, there are times things have to be coded a certain way).

Testability: Coverage tools like Clover, Emma, Jcoverage can help on your unit testing side to see if you can increase the amount of testing, and catch certain flow-changes (if/else/case/etc) in the code that aren't tested as well right now.

Step 4: The in-deep stuff
Once you reached step 4, you can look at the items I listed back at the top of this page. Notice that I really didn't hit certain items --

Profiling is actually an invasive process most of the time, I haven't found a tool that can easily identify memory issues or performance bottlenecks *for* you, instead they, rightfully, require you to review the information and come to your own conclusions. Tools like TPTP, Jprobe, jprofiler, etc aren't quick-fix tools, you need to learn them and understand them, and are useful in different scenarios.

Multi-tier profiling/review: Tracking from the Web tier, through the server tier (rules/workflow/business logic), to the database tier (sql/db, or sproc) to help identify where in the tiers a particular slowdown or issue is occuring from. Not something easy to do - some tools, like deprecated InfraRED and Glassbox, attempted to make this easier for us, but they don't seem to be active.

Integration testing: Actually being able to do business testing across an entire integrated system programatically, performance testing SOA or the full cycle of pressing a button in the UI are all desireable goals, but not easy to setup and do.

Automated UAT/User Interface testing: Some neat tools, like Selenium, can help step through testing a website and ensure things continue to work as expected. It's a great tool, but if you are constantly making changes, keeping those Selenium tests up to date can get time consuming. Also, need to know they do NOT identify blemishes/non-intuitive interface, only that the interface is continuing to work as expected.

Scalability testing: testing 10-years later equivalent worth of data, testing 5,10,50, 1000 concurrent users, evaluating estimate load ability per setup (proxy/multi-app servers/single db, db clustering, etc). You can also throw in disaster/recovery as part of the scalability testing as well. These are all very manual, very concious development purposes and is definitely invasive and time consuming.


Conclusion
Well, this looks more like a brain-dump than an organized blog, but sometimes just dropping information can be helpful to other people, and could solicit useful feedback!

Post-Edits:
A good article on related subject: http://www.ddj.com/architect/184415470

GWT 1.6 jvm crashes (ParameterizedMethodBinding)

Very quick blog. GWT 1.6 causing JVM crashes.

Sources:
http://grack.com/blog/2009/04/14/gwt-16-crashes-and-a-fix/

http://www.mail-archive.com/google-web-toolkit-contributors@googlegroups.com/msg04852.html

http://osdir.com/ml/GoogleWebToolkitContributors/2009-04/msg00044.html

We were experiencing similar issues, but the issues were difficult to sort out. It appeared Sun 1.5 JVM's did not have this issue, most of the Sun 1.6 JVM's did (but a newer one did not), and the OpenJDK 1.6.0-b09 also had the issue (issues spanning Windows and Linux boxes).

In the end, for our shop, I found the true resolution to the problem to be:

Remove the '-server' option when running the GWT compiler.

Microsoft, Open Source, Barriers to Entry/Barriers to Deploy

Microsoft, Open Source

I had to make a concious decision to have the above title have a comma between Microsoft and Open Source. Putting something like 'and' or 'versus' or 'with' may set the wrong stage for the intent of this blog.

links:
http://www.microsoft.com/opensource/
http://www.codeplex.com/


Focus:

• Barriers to Entry for a development environment
• Barriers to Deploy a solution

Traditionally, 'open source' is associated directly with programming languages, with the two more prominent (not only, just prominent) ones being Perl and Java.

So what makes them successful? A number of things do, but I wanted to mention two, from a business and adoption standpoint, key drivers -- Barriers to Entry and Barriers to Deploy.

Rather than have a lengthy paragraph, just going to bullet/summerize (BE = Barrier to Entry removed, BD = Barrier to Deploy removed):

Perl

• BE: You can get perl language for free.
• BE: You can get various IDE's to use perl for free...or use notepad/vi.
• BE: Large number of commercial books out there for Perl.
• BE: Large number of free articles out there for Perl.
• BE, BD: CPAN, a large centralized repository of code you can use, learn from, and deploy at will for free. A lot of problems, but obvious and obscure, have already been solved and are free for you to use and/or modify.
• BE, BD: Interpreted, can make changes on-the-fly and immediately see the results (good for learning and prototyping and fast support, questionable for enterprise apps).
  
• BD: Multi-OS environment support (with availability of free OS as deployment environment).
  

Java

• BE: You can get a java compiler and java VM without cost. Also, several options of compilers and VM's.
• BE: Eclipse/Netbeans IDE are free.
• BE: Large number of commercial books out there for Java.
• BE: Large number of college courses and training classes for Java (varying level of quality however).
• BE: Large number of free articles out there for Java, with code examples.
• BE, BD: Several tested/documented solution paths and design patterns for more complex solutions (OSGi, Spring, JavaEE).
• BE, BD: Many repositories of code and binaries available, free to use and modify -- sourceforge, codehaus, java.net, as well as Maven library repositories.
• BE, BD: Free to use and deploy build systems (ant, Maven) that are not tied to an IDE, and allows anyone to 'check out or download' code and just start working with it.
  
• BD: Multi-OS environment support (with availability of free OS as deployment environment).
• BD: Java has several servers (JavaEE container servers - Tomcat, Jetty, Jboss, Jonas, Glassfish, Geronimo, etc) that are also free to develop and deploy on.
• BE,BD: The JCP and/or common solutions usually have competition that continues innovation, and gives developers choices depending on the scenarios presented to them.

Microsoft

• BE: Large number of commerical books on Microsoft .NET programming language platforms.
• BE: Large number of college and training courses (relatively stable quality).
• BE: Various programming language options for the .NET platform.
• BE: Commerical MSDN access as a repository of solutions, code examples, etc.
• BE,BD: Graphical/UI builds through the singular, commericial (which is both good and bad, as it's a constant) IDE - Visual Studio.
• BD: You know exactly where it is going to deploy - commerical MS Server OS on MS IIS/Biztalk/etc licensed servers.
• BE: Lot of packaged solutions, some integration solutions, all commercial, are available. However, require licensing for deployments.
  

If Microsoft is going to try to adopt an open source community, they need to take a look at the Barriers to Entry and the Barriers to Deploy, particuarly from the commercial standpoint -- the companies that can spend the money aren't going to give their code back for free, while companies/developers that have low costs for the development and deployment environment are less in a pinch and like having their source code out there to help improve it's quality, particularly when the Barrier to Entry for someone else to look at published code is low.

Database Change Management

Preface: I am not a DBA. I'm a software developer, document imaging architect, build system and continuous integration implementer. I use databases to store, query, and retrieve information, and I'm a bigot against stored procedures (unless used as a performance enhancement tool with a non-sproc version already implemented, or it really is too complicated to do outside a sproc).

For those curious, I promised several people I would write up an open-source Data Profiler comparison blog, but I do not feel confident I have put in enough time for this yet.


Open Source Database Change Management

There isn't a lot out there.

http://code.google.com/p/dbdeploy/ - LGPL
http://www.liquibase.org/ - LGPL
http://opendbcopy.sourceforge.net/ - INACTIVE

edit: http://www.sql-workbench.net/ - WbSchemaDiff, WbDataDiff commandline

One of the wishlist items here: http://dhartford.blogspot.com/2008/11/all-i-want-for-christmas-2008-full.html

* Both the active are full featured.
* Both will handle change management well.
* Liquibase seems to have a larger integration-point base (maven, eclipse, etc)
* Neither have an adequate/sufficient UI that would be considered 'quality and usable by a DBA/non-programmer'. Yes, I'm a programmer, but I don't want to program EVERYTHING unless there is a reason for flexibility/customizability.

That's about all I have to say about this subject. The commercial tools still have a strong presence here (i.e. RedGate), as there aren't many alternatives.

For now, I still use old school database viewer/sql tools to extract the DDL and data value changes. The tool I use a lot is http://www.sql-workbench.net/ for this functionality, but I do hope for the above to mature and improve.

All I want for Christmas 2008 - Full Featured Eclipse Database plugin

Full Featured Database Plugin for Eclipse.

Definition is vague, and of course dependent on whom you talk to. I, personally, am more of a developer than a DBA, so you could instead call what I'm asking for as a 'Developer Database Plugin' versus a 'DBA Database Plugin'.

So why not use SQLexplorer, DTP, Clay, or the several other Eclipse plugins that, yes, I have tested/used?

Because, they are missing:

*Good Source Control Support
*Stored Procedure Support
*Simplified View Support
*Easy multi database support

Good Source Control Support (specifically Subversion)

For me, I immediately think of us spoiled Eclipse programmers that can:

1) See quickly on the navigation tree what items are different than what is in SCM (3 POVs, only worried about what is deployed to the DB server you are pointing to, and what is in SCM. Active Desktop changes should be deployed to the DB server you are pointing to first, then team-shared to SCM if all goes well).

2) Compare/Diff easily/visually between what is in SCM versus what I'm looking at now. I understand the tricky part is there is 3 POVs -- SCM, active desktop, and what is deployed to the server you are pointing to. From my standpoint, you always pull from the 'deployed db server' environment, compare to SCM, then make changes and deploy back to the 'deployed db server'.

3) Everything in SCM. Everything. Entire schema -- tables, stored procedures, functions, indexes, triggers, etc -- everything. Database specific security will be a bit trickier from an 'abstract' database standpoint, so that's ok, but no excuse on anything else.

Stored Procedure Support

This is so blatantly obvious it scares the heck out of me how many plugins don't support this. If it can be read from JDBC, or, more precisely, can be read via 'EXEC ' that could be a simple project to abstract out regardless of underlying database, it should be available from a plugin.

some caveats though --
1) Whatever mechanism is used to retrieve the deployed stored procedures, the -content- should be the same as what is stored to the SCM for easy compare/diff.

2) Keep the editors simple first. This means don't worry about it being t-sql, pl-sql, java, pgsql, etc -- just use simple text editors first, then work on developing more robust editors depending on the native stored procedure language of the underlying database. Work on it later....get this working for now first.


Simplified View Support

I'm not even going to bullet/number this. How many people raise your hands when you had to use DTP to drill down from the Database->Catalog->Schema-> when you thought you had pre-defined those values already in the connection. I'm all for flexibility, but tools like IDE and IDE Plugins are supposed to be designed for you to work effectively, so give us the ability to just look at what we want to look at (even if we have to configure it first).

Easy multi database support

Yes, we all know you can support any database that has a JDBC driver (one case). Yes we know there are specific features (Execution Plans, etc) that require custom code/integration/libraries for specific databases (second use case). But could you make it a little easier to get setup?

1) http://mirrors.ibiblio.org/pub/mirrors/maven2/ --- If you don't want to include the JDBC driver in your distribution, directly point your app to here under the right group id (directory structure) and let the person through your UI just pick a version and automatically download it to the right location for your app. Manually downloading and adding to the /lib, or classpath, is so old-school.

2) see #1. Dont be old school.

3) I really do like seeing more advanced tools like Execution plans, but don't let that be your primary focus unless you are running out of bugs/features. 80% of developers working with databases need all this other 80% functionality first.


Data Modelling

-1) This is not my list above.
-2) I actually enjoy Data Modelling, but unfortunately when it comes to hitting the ground running, sometimes it gets in the way and you need as direct access to the actual database you are working with (the specific MS SQL, specific Oracle, specific MySQL, etc...) to get work done.
-3) Triggers/sprocs with modelling...yeah...
-4) There are some pretty decent Data Modelling out there, so no need to solve a problem that already has several implementations -- but my above list really haven't found a solution yet!



Update: I'm looking at liquibase.org for SCM support, but not sure if I can squeeze in enough time to give them a good test run.

Eclipse on Fedora 9

If you are reading this post, more than likely you already know what I'm talking about.

Fedora 9 comes with their GCJ compiled version of 'Fedora Eclipse'. Nice idea, not well implemented. Trying to use Eclipse update sites do not work correctly, and/or want to use different eclipse distributions.

So, using the normal Eclipse.org distribution, or a custom distribution from various vendors, one would just download, untar/unzip, and run, right?

Wrong...you get the splash screen, loading modules, then a small grey box. Going to /.metadata/.log shows errors like this:

!ENTRY org.eclipse.ui.workbench 4 0 2008-10-14 15:35:07.364
!MESSAGE Widget disposed too early!
!STACK 0
java.lang.RuntimeException: Widget disposed too early!
at org.eclipse.ui.internal.WorkbenchPartReference$1.widgetDisposed(WorkbenchPartReference.java:171)
at org.eclipse.swt.widgets.TypedListener.handleEvent(TypedListener.java:117)
at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:84)

.....
org.eclipse.swt.SWTError: XPCOM error -2147467262
at org.eclipse.swt.browser.Mozilla.error(Mozilla.java:1638)
at org.eclipse.swt.browser.Mozilla.setText(Mozilla.java:1861)
at org.eclipse.swt.browser.Browser.setText(Browser.java:737)
at org.eclipse.ui.internal.intro.impl.presentations.BrowserIntroPartImplementation.generateContentForPage(BrowserIntroPartImplementation.java:252)
at org.eclipse.ui.internal.intro.impl.presentations.BrowserIntroPartImplementation.dynamicStandbyStateChanged(BrowserIntroPartImplementation.java:451)
at org.eclipse.ui.internal.intro.impl.presentations.BrowserIntroPartImplementation.doStandbyStateChanged(BrowserIntroPartImplementation.java:658)

The fix is simple...once you know what to fix. The last part, the Mozilla error, was the key. Not intuitive, but this is what you do to get an external Eclipse distribution to work on Fedora 9 with Sun or OpenJDK (not GCJ):

yum upgrade firefox

Yup, that's it. Just upgrade your Firefox install.

Freemarker CamelCase to underscore

Quick blog -- as always, the last 20% usually takes up 80% of the time.

This time, it was trying to simply convert Camel Case into equivalent underscore Enum values.

Ok, not that 'simply', but still -- I'm using Hibernate Tools to reverse engineer from JDBC some JPA entities, and that part is working fine. Now, UI and some processes prefer to use a Model that is on top of the entity/dto. So, I thought I would be nice and auto-generate the Model's that some other programmers swear by to make their job easier.

Hibernate Tools just went to FreeMarker, which I was excited for, and I wrote most of the .ftl up for my Model. Until I hit Camel Case.

You see, what they are trying to do is create an ENUM version of each field; I'm not going into detail why, but simply that code-generation wise --

fieldOne -> FIELD_ONE
myReallyLongComboField -> MY_REALLY_LONG_COMBO_FIELD

After a lot of messing around in Freemarker and regular expressions, finally got the solution in two lines in the .ftl file (very important the < /#macro> is where it is now):

<#macro toUnderScore camelCase>
${camelCase?replace("[A-Z]", "_$0", 'r')?upper_case}< /#macro>

Then, make calls like:

<@toUnderScore camelCase=property.name/>

Perfect!


I've had quite a bit of experience in the past with Velocity, and put in some work in code-gen tools like Middlegen (now defunct). Once you have a process/template for commonly used code pieces, code generation really helps enforce consistency and good practice.

EJB3 Seam GWT - part II, GWT 1.5m2

Short blog --
Working with Seam 2.1-SNAPSHOT and GWT 1.5M2 (with EJb3/jboss 4.2 backend).

To start off with a bang - JPA (hibernate) annotated beans are working!!! Yes, you can now use annotated entity beans as regular POJO's with Seam and GWT.

p.s. I really do *not* like the annotation approach and preferred the orm.xml deployment descriptor approach to keep the POJO truly clean and free from decoration with annotation behaviours -- but there are some problems with the hibernate orm.xml implementation that they 'lack resources' to correct, so whatever; use the actually supported annotation approach.

I unfortunately still not sure how to post code to blogger (need a full project, just sharing snippets is insufficient), but before you start jumping in too far here are the caveats:

• Long and Date fields do not work as of 5/1/2008. GWT changed the behaviour of Longs in Milestone 2 of GWT 1.5, and the Seam guys haven't modified to support this yet -- it is, afterall, still a milestone release of GWT. http://jira.jboss.org/jira/browse/JBSEAM-2935
• Arrays do not work as of 5/1/2008. This confuses me - Seam 2.0.0.GA and GWT 1.4.60 arrays worked fine, not sure why this is now broken. http://jira.jboss.org/jira/browse/JBSEAM-2933
• Lists (genericized) now do work, and can be used as an alternative to the broken array implementation.
• You can implement entitybean/pojo/DTO's that use the Long and Date fields, but when you RPC those fields you will get errors. You can test with your DTO and List as long as you do not actually use Long and Date fields (leave them blank/null).

Someone reminded me about something I've taken for granted - no, I am not using JNDI lookups. This is direct, session bean with Seam annotation exposed directly with the Seam-GWT remoting implementation to the GWT client. No middle JSF layer or JNDI layer or tomcat/servlet layer -- GWT web talking to EJB3/Seam server.

Still working on a more full example, have fun till then!

EJB3 Seam GWT

I just finished a prototype to test EJB3, Seam 2.0.0.GA with GWT-Remoting, and GWT 1.4.60.

It was, for the most part, a success!

First, before people get excited, what does NOT work:

• DTO's still can not be annotated, so no entity beans.
• Seam/GWT remoting integration requires the DTO's to implement google's 'IsSerializable'. http://jira.jboss.com/jira/browse/JBSEAM-2305

What has been proven to work:

• Stateless Session Bean simply exposed for GWT Web Application consumption for regular types (String, Integer, Long, Float) both as returns and parameters.
• SLSB with DTO's exposed for GWT Web Application consumption.
• GWT Sample Web Application successfully presenting the results of the Service calls to associated SLSB's exposed for Seam-GWT remoting.

Caveat - I am no GWT expert by any means. This project is simply a prototype to proof the integration points between existing EJB3 applications and changing the UI presentation layer to GWT in the least-difficult manner.

HELP: How do I share my project through blogger.com? For now, you can click on the JIRA link and get the uploaded file there - read the readme.txt and modify the Model for IsSerializable.

Digital Preservation - PST outlook files

I will probably have a big rant about Digital Preservation some day, but today just about personal files - the Outlook PST files.

First - why Outlook PST files, why not KMail/Thunderbird/Netscape/Sun/Whatever mail files? Well, quite frankly you will be hard pressed to find a business in the United States that does not use Outlook. One of those necessary evils as there still is no good open source PIM (i.e. email, contact, AND calendar) desktop tools that are DEPLOYABLE in a corporate environment.

Second - This is more a memo to myself when I have time, have not fully gone down this.

The primary concern I had with PST files was, well, they are proprietary. I want to fix that, and would prefer to be able to re-organize the many, many PST files and related e-mail entries I have (including, I'm sure, many duplicate email entries in different PST files).

-- Change from PST to something not PST.
* http://alioth.debian.org/projects/libpst/ - GPL, in C
* http://xena.sourceforge.net/index.html - GPL, in Java (still active, ODF conversions)

[XML output] As you can see by the sidenote, I'm leaning towards the ill-named Xena project as it is 1) still active, 2) in java, and 3) *may* be able to export in ODF. I say may, because it doesn't say it specifically regarding e-mail.

[mbox] The other, libpst, will convert the PST into a unix-style mbox format.

[maildir] Maildir would have been my preference, but with the ODF being a very close second. However, the only maildir open source export I could find was http://www.howtoforge.com/converting_outlook_pst_to_maildir, and this required PUTTING THE MAIL BACK ON THE SERVER TO RE-READ IT BACK THROUGH IMAP. Very cool for going-forward projects, not so much as a library to do a simple convert. And, any sysAdmin would have a fit if I put 5gig of PST files back onto an Exchange server.

Managing -
Absolutely nothing yet. Order, remove duplicates, partition to put into CD/DVD media, a simple stand-alone client that can be put on a CD/DVD to read the archived e-mails, etc.

Please comment if you have found an open-source approach to solving PST archival and management.

Humans and Content - Information

This post is based on reading an article on the increase in time people spend reading content. The article link is here: http://news.yahoo.com/s/nm/20070813/tc_nm/internet_study_dc_1

Now, my own personal opinion is that video content is very time consuming. Although it may be the easiest to digest, it is the most inefficient and getting information. Most of this blog is about getting information, not about entertainment purposes.

If you are reading this blog -- you probably speed-reading and just skipping most of the fluff to get some key pieces of information. That is good -- that is how it is suppose to be.

Video/audio content, however, you do not have that option. Written works have no time component, only an 'order of content' component that you can filter through quickly. However,
Video/audio content DOES have a time component meaning you are limited in efficiently digesting/absorbing information based on the time component established by whoever created the Video/audio.

I personally think podcasts/webcasts are one of the worse ways to send out information (good for advertising/entertainment, but poor for sharing information). I pretty much never watch a podcast/webcast for information -- instead I'll Google for a text version and read through that instead.

Now, that is just me - there are people/audience that do prefer video/audio content for getting information. And, I might be one of those people...if I didn't have to worry about time ;-)