Very quick blog. GWT 1.6 causing JVM crashes.
Sources:
http://grack.com/blog/2009/04/14/gwt-16-crashes-and-a-fix/
http://www.mail-archive.com/google-web-toolkit-contributors@googlegroups.com/msg04852.html
http://osdir.com/ml/GoogleWebToolkitContributors/2009-04/msg00044.html
We were experiencing similar issues, but the issues were difficult to sort out. It appeared Sun 1.5 JVM's did not have this issue, most of the Sun 1.6 JVM's did (but a newer one did not), and the OpenJDK 1.6.0-b09 also had the issue (issues spanning Windows and Linux boxes).
In the end, for our shop, I found the true resolution to the problem to be:
Remove the '-server' option when running the GWT compiler.
Thursday, July 23, 2009
Wednesday, July 01, 2009
Microsoft, Open Source, Barriers to Entry/Barriers to Deploy
Microsoft, Open Source
I had to make a concious decision to have the above title have a comma between Microsoft and Open Source. Putting something like 'and' or 'versus' or 'with' may set the wrong stage for the intent of this blog.
links:
http://www.microsoft.com/opensource/
http://www.codeplex.com/
Focus:
Traditionally, 'open source' is associated directly with programming languages, with the two more prominent (not only, just prominent) ones being Perl and Java.
So what makes them successful? A number of things do, but I wanted to mention two, from a business and adoption standpoint, key drivers -- Barriers to Entry and Barriers to Deploy.
Rather than have a lengthy paragraph, just going to bullet/summerize (BE = Barrier to Entry removed, BD = Barrier to Deploy removed):
Perl
Microsoft
I had to make a concious decision to have the above title have a comma between Microsoft and Open Source. Putting something like 'and' or 'versus' or 'with' may set the wrong stage for the intent of this blog.
links:
http://www.microsoft.com/opensource/
http://www.codeplex.com/
Focus:
- Barriers to Entry for a development environment
- Barriers to Deploy a solution
Traditionally, 'open source' is associated directly with programming languages, with the two more prominent (not only, just prominent) ones being Perl and Java.
So what makes them successful? A number of things do, but I wanted to mention two, from a business and adoption standpoint, key drivers -- Barriers to Entry and Barriers to Deploy.
Rather than have a lengthy paragraph, just going to bullet/summerize (BE = Barrier to Entry removed, BD = Barrier to Deploy removed):
Perl
- BE: You can get perl language for free.
- BE: You can get various IDE's to use perl for free...or use notepad/vi.
- BE: Large number of commercial books out there for Perl.
- BE: Large number of free articles out there for Perl.
- BE, BD: CPAN, a large centralized repository of code you can use, learn from, and deploy at will for free. A lot of problems, but obvious and obscure, have already been solved and are free for you to use and/or modify.
- BE, BD: Interpreted, can make changes on-the-fly and immediately see the results (good for learning and prototyping and fast support, questionable for enterprise apps).
- BD: Multi-OS environment support (with availability of free OS as deployment environment).
- BE: You can get a java compiler and java VM without cost. Also, several options of compilers and VM's.
- BE: Eclipse/Netbeans IDE are free.
- BE: Large number of commercial books out there for Java.
- BE: Large number of college courses and training classes for Java (varying level of quality however).
- BE: Large number of free articles out there for Java, with code examples.
- BE, BD: Several tested/documented solution paths and design patterns for more complex solutions (OSGi, Spring, JavaEE).
- BE, BD: Many repositories of code and binaries available, free to use and modify -- sourceforge, codehaus, java.net, as well as Maven library repositories.
- BE, BD: Free to use and deploy build systems (ant, Maven) that are not tied to an IDE, and allows anyone to 'check out or download' code and just start working with it.
- BD: Multi-OS environment support (with availability of free OS as deployment environment).
- BD: Java has several servers (JavaEE container servers - Tomcat, Jetty, Jboss, Jonas, Glassfish, Geronimo, etc) that are also free to develop and deploy on.
- BE,BD: The JCP and/or common solutions usually have competition that continues innovation, and gives developers choices depending on the scenarios presented to them.
Microsoft
- BE: Large number of commerical books on Microsoft .NET programming language platforms.
- BE: Large number of college and training courses (relatively stable quality).
- BE: Various programming language options for the .NET platform.
- BE: Commerical MSDN access as a repository of solutions, code examples, etc.
- BE,BD: Graphical/UI builds through the singular, commericial (which is both good and bad, as it's a constant) IDE - Visual Studio.
- BD: You know exactly where it is going to deploy - commerical MS Server OS on MS IIS/Biztalk/etc licensed servers.
- BE: Lot of packaged solutions, some integration solutions, all commercial, are available. However, require licensing for deployments.
Thursday, June 11, 2009
Database Change Management
Preface: I am not a DBA. I'm a software developer, document imaging architect, build system and continuous integration implementer. I use databases to store, query, and retrieve information, and I'm a bigot against stored procedures (unless used as a performance enhancement tool with a non-sproc version already implemented, or it really is too complicated to do outside a sproc).
For those curious, I promised several people I would write up an open-source Data Profiler comparison blog, but I do not feel confident I have put in enough time for this yet.
Open Source Database Change Management
There isn't a lot out there.
http://code.google.com/p/dbdeploy/ - LGPL
http://www.liquibase.org/ - LGPL
http://opendbcopy.sourceforge.net/ - INACTIVE
edit: http://www.sql-workbench.net/ - WbSchemaDiff, WbDataDiff commandline
One of the wishlist items here: http://dhartford.blogspot.com/2008/11/all-i-want-for-christmas-2008-full.html
* Both the active are full featured.
* Both will handle change management well.
* Liquibase seems to have a larger integration-point base (maven, eclipse, etc)
* Neither have an adequate/sufficient UI that would be considered 'quality and usable by a DBA/non-programmer'. Yes, I'm a programmer, but I don't want to program EVERYTHING unless there is a reason for flexibility/customizability.
That's about all I have to say about this subject. The commercial tools still have a strong presence here (i.e. RedGate), as there aren't many alternatives.
For now, I still use old school database viewer/sql tools to extract the DDL and data value changes. The tool I use a lot is http://www.sql-workbench.net/ for this functionality, but I do hope for the above to mature and improve.
For those curious, I promised several people I would write up an open-source Data Profiler comparison blog, but I do not feel confident I have put in enough time for this yet.
Open Source Database Change Management
There isn't a lot out there.
http://code.google.com/p/dbdeploy/ - LGPL
http://www.liquibase.org/ - LGPL
http://opendbcopy.sourceforge.net/ - INACTIVE
edit: http://www.sql-workbench.net/ - WbSchemaDiff, WbDataDiff commandline
One of the wishlist items here: http://dhartford.blogspot.com/2008/11/all-i-want-for-christmas-2008-full.html
* Both the active are full featured.
* Both will handle change management well.
* Liquibase seems to have a larger integration-point base (maven, eclipse, etc)
* Neither have an adequate/sufficient UI that would be considered 'quality and usable by a DBA/non-programmer'. Yes, I'm a programmer, but I don't want to program EVERYTHING unless there is a reason for flexibility/customizability.
That's about all I have to say about this subject. The commercial tools still have a strong presence here (i.e. RedGate), as there aren't many alternatives.
For now, I still use old school database viewer/sql tools to extract the DDL and data value changes. The tool I use a lot is http://www.sql-workbench.net/ for this functionality, but I do hope for the above to mature and improve.
Wednesday, November 12, 2008
All I want for Christmas 2008 - Full Featured Eclipse Database plugin
Full Featured Database Plugin for Eclipse.
Definition is vague, and of course dependent on whom you talk to. I, personally, am more of a developer than a DBA, so you could instead call what I'm asking for as a 'Developer Database Plugin' versus a 'DBA Database Plugin'.
So why not use SQLexplorer, DTP, Clay, or the several other Eclipse plugins that, yes, I have tested/used?
Because, they are missing:
*Good Source Control Support
*Stored Procedure Support
*Simplified View Support
*Easy multi database support
Good Source Control Support (specifically Subversion)
For me, I immediately think of us spoiled Eclipse programmers that can:
1) See quickly on the navigation tree what items are different than what is in SCM (3 POVs, only worried about what is deployed to the DB server you are pointing to, and what is in SCM. Active Desktop changes should be deployed to the DB server you are pointing to first, then team-shared to SCM if all goes well).
2) Compare/Diff easily/visually between what is in SCM versus what I'm looking at now. I understand the tricky part is there is 3 POVs -- SCM, active desktop, and what is deployed to the server you are pointing to. From my standpoint, you always pull from the 'deployed db server' environment, compare to SCM, then make changes and deploy back to the 'deployed db server'.
3) Everything in SCM. Everything. Entire schema -- tables, stored procedures, functions, indexes, triggers, etc -- everything. Database specific security will be a bit trickier from an 'abstract' database standpoint, so that's ok, but no excuse on anything else.
Stored Procedure Support
This is so blatantly obvious it scares the heck out of me how many plugins don't support this. If it can be read from JDBC, or, more precisely, can be read via 'EXEC' that could be a simple project to abstract out regardless of underlying database, it should be available from a plugin.
some caveats though --
1) Whatever mechanism is used to retrieve the deployed stored procedures, the -content- should be the same as what is stored to the SCM for easy compare/diff.
2) Keep the editors simple first. This means don't worry about it being t-sql, pl-sql, java, pgsql, etc -- just use simple text editors first, then work on developing more robust editors depending on the native stored procedure language of the underlying database. Work on it later....get this working for now first.
Simplified View Support
I'm not even going to bullet/number this. How many people raise your hands when you had to use DTP to drill down from the Database->Catalog->Schema-> when you thought you had pre-defined those values already in the connection. I'm all for flexibility, but tools like IDE and IDE Plugins are supposed to be designed for you to work effectively, so give us the ability to just look at what we want to look at (even if we have to configure it first).
Easy multi database support
Yes, we all know you can support any database that has a JDBC driver (one case). Yes we know there are specific features (Execution Plans, etc) that require custom code/integration/libraries for specific databases (second use case). But could you make it a little easier to get setup?
1) http://mirrors.ibiblio.org/pub/mirrors/maven2/ --- If you don't want to include the JDBC driver in your distribution, directly point your app to here under the right group id (directory structure) and let the person through your UI just pick a version and automatically download it to the right location for your app. Manually downloading and adding to the /lib, or classpath, is so old-school.
2) see #1. Dont be old school.
3) I really do like seeing more advanced tools like Execution plans, but don't let that be your primary focus unless you are running out of bugs/features. 80% of developers working with databases need all this other 80% functionality first.
Data Modelling
-1) This is not my list above.
-2) I actually enjoy Data Modelling, but unfortunately when it comes to hitting the ground running, sometimes it gets in the way and you need as direct access to the actual database you are working with (the specific MS SQL, specific Oracle, specific MySQL, etc...) to get work done.
-3) Triggers/sprocs with modelling...yeah...
-4) There are some pretty decent Data Modelling out there, so no need to solve a problem that already has several implementations -- but my above list really haven't found a solution yet!
Update: I'm looking at liquibase.org for SCM support, but not sure if I can squeeze in enough time to give them a good test run.
Definition is vague, and of course dependent on whom you talk to. I, personally, am more of a developer than a DBA, so you could instead call what I'm asking for as a 'Developer Database Plugin' versus a 'DBA Database Plugin'.
So why not use SQLexplorer, DTP, Clay, or the several other Eclipse plugins that, yes, I have tested/used?
Because, they are missing:
*Good Source Control Support
*Stored Procedure Support
*Simplified View Support
*Easy multi database support
Good Source Control Support (specifically Subversion)
For me, I immediately think of us spoiled Eclipse programmers that can:
1) See quickly on the navigation tree what items are different than what is in SCM (3 POVs, only worried about what is deployed to the DB server you are pointing to, and what is in SCM. Active Desktop changes should be deployed to the DB server you are pointing to first, then team-shared to SCM if all goes well).
2) Compare/Diff easily/visually between what is in SCM versus what I'm looking at now. I understand the tricky part is there is 3 POVs -- SCM, active desktop, and what is deployed to the server you are pointing to. From my standpoint, you always pull from the 'deployed db server' environment, compare to SCM, then make changes and deploy back to the 'deployed db server'.
3) Everything in SCM. Everything. Entire schema -- tables, stored procedures, functions, indexes, triggers, etc -- everything. Database specific security will be a bit trickier from an 'abstract' database standpoint, so that's ok, but no excuse on anything else.
Stored Procedure Support
This is so blatantly obvious it scares the heck out of me how many plugins don't support this. If it can be read from JDBC, or, more precisely, can be read via 'EXEC
some caveats though --
1) Whatever mechanism is used to retrieve the deployed stored procedures, the -content- should be the same as what is stored to the SCM for easy compare/diff.
2) Keep the editors simple first. This means don't worry about it being t-sql, pl-sql, java, pgsql, etc -- just use simple text editors first, then work on developing more robust editors depending on the native stored procedure language of the underlying database. Work on it later....get this working for now first.
Simplified View Support
I'm not even going to bullet/number this. How many people raise your hands when you had to use DTP to drill down from the Database->Catalog->Schema-> when you thought you had pre-defined those values already in the connection. I'm all for flexibility, but tools like IDE and IDE Plugins are supposed to be designed for you to work effectively, so give us the ability to just look at what we want to look at (even if we have to configure it first).
Easy multi database support
Yes, we all know you can support any database that has a JDBC driver (one case). Yes we know there are specific features (Execution Plans, etc) that require custom code/integration/libraries for specific databases (second use case). But could you make it a little easier to get setup?
1) http://mirrors.ibiblio.org/pub/mirrors/maven2/ --- If you don't want to include the JDBC driver in your distribution, directly point your app to here under the right group id (directory structure) and let the person through your UI just pick a version and automatically download it to the right location for your app. Manually downloading and adding to the /lib, or classpath, is so old-school.
2) see #1. Dont be old school.
3) I really do like seeing more advanced tools like Execution plans, but don't let that be your primary focus unless you are running out of bugs/features. 80% of developers working with databases need all this other 80% functionality first.
Data Modelling
-1) This is not my list above.
-2) I actually enjoy Data Modelling, but unfortunately when it comes to hitting the ground running, sometimes it gets in the way and you need as direct access to the actual database you are working with (the specific MS SQL, specific Oracle, specific MySQL, etc...) to get work done.
-3) Triggers/sprocs with modelling...yeah...
-4) There are some pretty decent Data Modelling out there, so no need to solve a problem that already has several implementations -- but my above list really haven't found a solution yet!
Update: I'm looking at liquibase.org for SCM support, but not sure if I can squeeze in enough time to give them a good test run.
Monday, November 03, 2008
Eclipse on Fedora 9
If you are reading this post, more than likely you already know what I'm talking about.
Fedora 9 comes with their GCJ compiled version of 'Fedora Eclipse'. Nice idea, not well implemented. Trying to use Eclipse update sites do not work correctly, and/or want to use different eclipse distributions.
So, using the normal Eclipse.org distribution, or a custom distribution from various vendors, one would just download, untar/unzip, and run, right?
Wrong...you get the splash screen, loading modules, then a small grey box. Going to/.metadata/.log shows errors like this:
!ENTRY org.eclipse.ui.workbench 4 0 2008-10-14 15:35:07.364
!MESSAGE Widget disposed too early!
!STACK 0
java.lang.RuntimeException: Widget disposed too early!
at org.eclipse.ui.internal.WorkbenchPartReference$1.widgetDisposed(WorkbenchPartReference.java:171)
at org.eclipse.swt.widgets.TypedListener.handleEvent(TypedListener.java:117)
at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:84)
.....
org.eclipse.swt.SWTError: XPCOM error -2147467262
at org.eclipse.swt.browser.Mozilla.error(Mozilla.java:1638)
at org.eclipse.swt.browser.Mozilla.setText(Mozilla.java:1861)
at org.eclipse.swt.browser.Browser.setText(Browser.java:737)
at org.eclipse.ui.internal.intro.impl.presentations.BrowserIntroPartImplementation.generateContentForPage(BrowserIntroPartImplementation.java:252)
at org.eclipse.ui.internal.intro.impl.presentations.BrowserIntroPartImplementation.dynamicStandbyStateChanged(BrowserIntroPartImplementation.java:451)
at org.eclipse.ui.internal.intro.impl.presentations.BrowserIntroPartImplementation.doStandbyStateChanged(BrowserIntroPartImplementation.java:658)
The fix is simple...once you know what to fix. The last part, the Mozilla error, was the key. Not intuitive, but this is what you do to get an external Eclipse distribution to work on Fedora 9 with Sun or OpenJDK (not GCJ):
yum upgrade firefox
Yup, that's it. Just upgrade your Firefox install.
Fedora 9 comes with their GCJ compiled version of 'Fedora Eclipse'. Nice idea, not well implemented. Trying to use Eclipse update sites do not work correctly, and/or want to use different eclipse distributions.
So, using the normal Eclipse.org distribution, or a custom distribution from various vendors, one would just download, untar/unzip, and run, right?
Wrong...you get the splash screen, loading modules, then a small grey box. Going to
!ENTRY org.eclipse.ui.workbench 4 0 2008-10-14 15:35:07.364
!MESSAGE Widget disposed too early!
!STACK 0
java.lang.RuntimeException: Widget disposed too early!
at org.eclipse.ui.internal.WorkbenchPartReference$1.widgetDisposed(WorkbenchPartReference.java:171)
at org.eclipse.swt.widgets.TypedListener.handleEvent(TypedListener.java:117)
at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:84)
.....
org.eclipse.swt.SWTError: XPCOM error -2147467262
at org.eclipse.swt.browser.Mozilla.error(Mozilla.java:1638)
at org.eclipse.swt.browser.Mozilla.setText(Mozilla.java:1861)
at org.eclipse.swt.browser.Browser.setText(Browser.java:737)
at org.eclipse.ui.internal.intro.impl.presentations.BrowserIntroPartImplementation.generateContentForPage(BrowserIntroPartImplementation.java:252)
at org.eclipse.ui.internal.intro.impl.presentations.BrowserIntroPartImplementation.dynamicStandbyStateChanged(BrowserIntroPartImplementation.java:451)
at org.eclipse.ui.internal.intro.impl.presentations.BrowserIntroPartImplementation.doStandbyStateChanged(BrowserIntroPartImplementation.java:658)
The fix is simple...once you know what to fix. The last part, the Mozilla error, was the key. Not intuitive, but this is what you do to get an external Eclipse distribution to work on Fedora 9 with Sun or OpenJDK (not GCJ):
yum upgrade firefox
Yup, that's it. Just upgrade your Firefox install.
Monday, July 07, 2008
Freemarker CamelCase to underscore
Quick blog -- as always, the last 20% usually takes up 80% of the time.
This time, it was trying to simply convert Camel Case into equivalent underscore Enum values.
Ok, not that 'simply', but still -- I'm using Hibernate Tools to reverse engineer from JDBC some JPA entities, and that part is working fine. Now, UI and some processes prefer to use a Model that is on top of the entity/dto. So, I thought I would be nice and auto-generate the Model's that some other programmers swear by to make their job easier.
Hibernate Tools just went to FreeMarker, which I was excited for, and I wrote most of the .ftl up for my Model. Until I hit Camel Case.
You see, what they are trying to do is create an ENUM version of each field; I'm not going into detail why, but simply that code-generation wise --
fieldOne -> FIELD_ONE
myReallyLongComboField -> MY_REALLY_LONG_COMBO_FIELD
After a lot of messing around in Freemarker and regular expressions, finally got the solution in two lines in the .ftl file (very important the < /#macro> is where it is now):
<#macro toUnderScore camelCase>
${camelCase?replace("[A-Z]", "_$0", 'r')?upper_case}< /#macro>
Then, make calls like:
<@toUnderScore camelCase=property.name/>
Perfect!
I've had quite a bit of experience in the past with Velocity, and put in some work in code-gen tools like Middlegen (now defunct). Once you have a process/template for commonly used code pieces, code generation really helps enforce consistency and good practice.
This time, it was trying to simply convert Camel Case into equivalent underscore Enum values.
Ok, not that 'simply', but still -- I'm using Hibernate Tools to reverse engineer from JDBC some JPA entities, and that part is working fine. Now, UI and some processes prefer to use a Model that is on top of the entity/dto. So, I thought I would be nice and auto-generate the Model's that some other programmers swear by to make their job easier.
Hibernate Tools just went to FreeMarker, which I was excited for, and I wrote most of the .ftl up for my Model. Until I hit Camel Case.
You see, what they are trying to do is create an ENUM version of each field; I'm not going into detail why, but simply that code-generation wise --
fieldOne -> FIELD_ONE
myReallyLongComboField -> MY_REALLY_LONG_COMBO_FIELD
After a lot of messing around in Freemarker and regular expressions, finally got the solution in two lines in the .ftl file (very important the < /#macro> is where it is now):
<#macro toUnderScore camelCase>
${camelCase?replace("[A-Z]", "_$0", 'r')?upper_case}< /#macro>
Then, make calls like:
<@toUnderScore camelCase=property.name/>
Perfect!
I've had quite a bit of experience in the past with Velocity, and put in some work in code-gen tools like Middlegen (now defunct). Once you have a process/template for commonly used code pieces, code generation really helps enforce consistency and good practice.
Thursday, May 01, 2008
EJB3 Seam GWT - part II, GWT 1.5m2
Short blog --
Working with Seam 2.1-SNAPSHOT and GWT 1.5M2 (with EJb3/jboss 4.2 backend).
To start off with a bang - JPA (hibernate) annotated beans are working!!! Yes, you can now use annotated entity beans as regular POJO's with Seam and GWT.
p.s. I really do *not* like the annotation approach and preferred the orm.xml deployment descriptor approach to keep the POJO truly clean and free from decoration with annotation behaviours -- but there are some problems with the hibernate orm.xml implementation that they 'lack resources' to correct, so whatever; use the actually supported annotation approach.
I unfortunately still not sure how to post code to blogger (need a full project, just sharing snippets is insufficient), but before you start jumping in too far here are the caveats:
Someone reminded me about something I've taken for granted - no, I am not using JNDI lookups. This is direct, session bean with Seam annotation exposed directly with the Seam-GWT remoting implementation to the GWT client. No middle JSF layer or JNDI layer or tomcat/servlet layer -- GWT web talking to EJB3/Seam server.
Still working on a more full example, have fun till then!
Working with Seam 2.1-SNAPSHOT and GWT 1.5M2 (with EJb3/jboss 4.2 backend).
To start off with a bang - JPA (hibernate) annotated beans are working!!! Yes, you can now use annotated entity beans as regular POJO's with Seam and GWT.
p.s. I really do *not* like the annotation approach and preferred the orm.xml deployment descriptor approach to keep the POJO truly clean and free from decoration with annotation behaviours -- but there are some problems with the hibernate orm.xml implementation that they 'lack resources' to correct, so whatever; use the actually supported annotation approach.
I unfortunately still not sure how to post code to blogger (need a full project, just sharing snippets is insufficient), but before you start jumping in too far here are the caveats:
- Long and Date fields do not work as of 5/1/2008. GWT changed the behaviour of Longs in Milestone 2 of GWT 1.5, and the Seam guys haven't modified to support this yet -- it is, afterall, still a milestone release of GWT. http://jira.jboss.org/jira/browse/JBSEAM-2935
- Arrays do not work as of 5/1/2008. This confuses me - Seam 2.0.0.GA and GWT 1.4.60 arrays worked fine, not sure why this is now broken. http://jira.jboss.org/jira/browse/JBSEAM-2933
- Lists (genericized) now do work, and can be used as an alternative to the broken array implementation.
- You can implement entitybean/pojo/DTO's that use the Long and Date fields, but when you RPC those fields you will get errors. You can test with your DTO and List
as long as you do not actually use Long and Date fields (leave them blank/null).
Someone reminded me about something I've taken for granted - no, I am not using JNDI lookups. This is direct, session bean with Seam annotation exposed directly with the Seam-GWT remoting implementation to the GWT client. No middle JSF layer or JNDI layer or tomcat/servlet layer -- GWT web talking to EJB3/Seam server.
Still working on a more full example, have fun till then!
Wednesday, November 28, 2007
EJB3 Seam GWT
I just finished a prototype to test EJB3, Seam 2.0.0.GA with GWT-Remoting, and GWT 1.4.60.
It was, for the most part, a success!
First, before people get excited, what does NOT work:
HELP: How do I share my project through blogger.com? For now, you can click on the JIRA link and get the uploaded file there - read the readme.txt and modify the Model for IsSerializable.
It was, for the most part, a success!
First, before people get excited, what does NOT work:
- DTO's still can not be annotated, so no entity beans.
- Seam/GWT remoting integration requires the DTO's to implement google's 'IsSerializable'. http://jira.jboss.com/jira/browse/JBSEAM-2305
- Stateless Session Bean simply exposed for GWT Web Application consumption for regular types (String, Integer, Long, Float) both as returns and parameters.
- SLSB with DTO's exposed for GWT Web Application consumption.
- GWT Sample Web Application successfully presenting the results of the Service calls to associated SLSB's exposed for Seam-GWT remoting.
HELP: How do I share my project through blogger.com? For now, you can click on the JIRA link and get the uploaded file there - read the readme.txt and modify the Model for IsSerializable.
Thursday, August 23, 2007
Digital Preservation - PST outlook files
I will probably have a big rant about Digital Preservation some day, but today just about personal files - the Outlook PST files.
First - why Outlook PST files, why not KMail/Thunderbird/Netscape/Sun/Whatever mail files? Well, quite frankly you will be hard pressed to find a business in the United States that does not use Outlook. One of those necessary evils as there still is no good open source PIM (i.e. email, contact, AND calendar) desktop tools that are DEPLOYABLE in a corporate environment.
Second - This is more a memo to myself when I have time, have not fully gone down this.
The primary concern I had with PST files was, well, they are proprietary. I want to fix that, and would prefer to be able to re-organize the many, many PST files and related e-mail entries I have (including, I'm sure, many duplicate email entries in different PST files).
-- Change from PST to something not PST.
* http://alioth.debian.org/projects/libpst/ - GPL, in C
* http://xena.sourceforge.net/index.html - GPL, in Java (still active, ODF conversions)
[XML output] As you can see by the sidenote, I'm leaning towards the ill-named Xena project as it is 1) still active, 2) in java, and 3) *may* be able to export in ODF. I say may, because it doesn't say it specifically regarding e-mail.
[mbox] The other, libpst, will convert the PST into a unix-style mbox format.
[maildir] Maildir would have been my preference, but with the ODF being a very close second. However, the only maildir open source export I could find was http://www.howtoforge.com/converting_outlook_pst_to_maildir, and this required PUTTING THE MAIL BACK ON THE SERVER TO RE-READ IT BACK THROUGH IMAP. Very cool for going-forward projects, not so much as a library to do a simple convert. And, any sysAdmin would have a fit if I put 5gig of PST files back onto an Exchange server.
Managing -
Absolutely nothing yet. Order, remove duplicates, partition to put into CD/DVD media, a simple stand-alone client that can be put on a CD/DVD to read the archived e-mails, etc.
Please comment if you have found an open-source approach to solving PST archival and management.
First - why Outlook PST files, why not KMail/Thunderbird/Netscape/Sun/Whatever mail files? Well, quite frankly you will be hard pressed to find a business in the United States that does not use Outlook. One of those necessary evils as there still is no good open source PIM (i.e. email, contact, AND calendar) desktop tools that are DEPLOYABLE in a corporate environment.
Second - This is more a memo to myself when I have time, have not fully gone down this.
The primary concern I had with PST files was, well, they are proprietary. I want to fix that, and would prefer to be able to re-organize the many, many PST files and related e-mail entries I have (including, I'm sure, many duplicate email entries in different PST files).
-- Change from PST to something not PST.
* http://alioth.debian.org/projects/libpst/ - GPL, in C
* http://xena.sourceforge.net/index.html - GPL, in Java (still active, ODF conversions)
[XML output] As you can see by the sidenote, I'm leaning towards the ill-named Xena project as it is 1) still active, 2) in java, and 3) *may* be able to export in ODF. I say may, because it doesn't say it specifically regarding e-mail.
[mbox] The other, libpst, will convert the PST into a unix-style mbox format.
[maildir] Maildir would have been my preference, but with the ODF being a very close second. However, the only maildir open source export I could find was http://www.howtoforge.com/converting_outlook_pst_to_maildir, and this required PUTTING THE MAIL BACK ON THE SERVER TO RE-READ IT BACK THROUGH IMAP. Very cool for going-forward projects, not so much as a library to do a simple convert. And, any sysAdmin would have a fit if I put 5gig of PST files back onto an Exchange server.
Managing -
Absolutely nothing yet. Order, remove duplicates, partition to put into CD/DVD media, a simple stand-alone client that can be put on a CD/DVD to read the archived e-mails, etc.
Please comment if you have found an open-source approach to solving PST archival and management.
Monday, August 13, 2007
Humans and Content - Information
This post is based on reading an article on the increase in time people spend reading content. The article link is here: http://news.yahoo.com/s/nm/20070813/tc_nm/internet_study_dc_1
Now, my own personal opinion is that video content is very time consuming. Although it may be the easiest to digest, it is the most inefficient and getting information. Most of this blog is about getting information, not about entertainment purposes.
If you are reading this blog -- you probably speed-reading and just skipping most of the fluff to get some key pieces of information. That is good -- that is how it is suppose to be.
Video/audio content, however, you do not have that option. Written works have no time component, only an 'order of content' component that you can filter through quickly. However,
Video/audio content DOES have a time component meaning you are limited in efficiently digesting/absorbing information based on the time component established by whoever created the Video/audio.
I personally think podcasts/webcasts are one of the worse ways to send out information (good for advertising/entertainment, but poor for sharing information). I pretty much never watch a podcast/webcast for information -- instead I'll Google for a text version and read through that instead.
Now, that is just me - there are people/audience that do prefer video/audio content for getting information. And, I might be one of those people...if I didn't have to worry about time ;-)
Now, my own personal opinion is that video content is very time consuming. Although it may be the easiest to digest, it is the most inefficient and getting information. Most of this blog is about getting information, not about entertainment purposes.
If you are reading this blog -- you probably speed-reading and just skipping most of the fluff to get some key pieces of information. That is good -- that is how it is suppose to be.
Video/audio content, however, you do not have that option. Written works have no time component, only an 'order of content' component that you can filter through quickly. However,
Video/audio content DOES have a time component meaning you are limited in efficiently digesting/absorbing information based on the time component established by whoever created the Video/audio.
I personally think podcasts/webcasts are one of the worse ways to send out information (good for advertising/entertainment, but poor for sharing information). I pretty much never watch a podcast/webcast for information -- instead I'll Google for a text version and read through that instead.
Now, that is just me - there are people/audience that do prefer video/audio content for getting information. And, I might be one of those people...if I didn't have to worry about time ;-)
Wednesday, June 20, 2007
Carnal Knowledge API
Quite the title, eh?
This post is about API, services, or interfaces that are obscure and require 'internal' knowledge to use successfully. What do I mean?
Object result = doIt(object1);
There are two specific scenarios that I think about for obscure AI/services:
*Carnal Modification
*Carnal Returns
Carnal Modification
This happens only in API's where the language allows passing of references and the objects passed are non-immutable.
System.out.println(bean1.getValue()); //prints "default"
void modifyJavabeanValue(bean1);
System.out.println(bean1.getValue()); //prints "modified"
By simply calling a method, the objects you passed to it have changed. This may not be an expected result and you have to know that is the intent of the API...i.e., you have to have carnal knowledge about it. And, do not be fooled if it has a return-type, it can still modify the reference!
Carnal Returns
Carnal returns requires significant pre-knowledge on how to handle the return.
Object o = getMyStuff();
In the above example, you have no idea what is supposed to be returned, and even worse, it may return one of, say, five different types of objects that do not have common interfaces. Although you can check/reflect (pending language) what the actual object-type is supposed to be. Horrible!!!
String result = changeThis(String rawdata);
This example is almost as bad - the returned String content may be something unexpected: i.e., could be XML, could be comma-delimited string, could be raw java/perl/php code that you are expected to run. This can be allieviated easily with documentation AND specifying in the method signature the expected result:
String result = changeThisToXML(String rawdata); //returns XML
Awareness
Just trying to share some awareness that just because you found a neat/cool way to pull something off, other people (or you using someone elses) may run into obscure or unexpected results related to Carnal Knowledge requirements. There are indeed times when you can only do it a certain way, just rememer to document and modify your method signatures to make it as clear as possible -- you never know, 5 years later you might have to use your own API/Service!
NEW: I recently learned that, surprisingly, there is functionality when writing Stored Procedures to *change* the fields in the resultset based on parameters passed in...and that people do this!! Exact same problem.
This post is about API, services, or interfaces that are obscure and require 'internal' knowledge to use successfully. What do I mean?
Object result = doIt(object1);
There are two specific scenarios that I think about for obscure AI/services:
*Carnal Modification
*Carnal Returns
Carnal Modification
This happens only in API's where the language allows passing of references and the objects passed are non-immutable.
System.out.println(bean1.getValue()); //prints "default"
void modifyJavabeanValue(bean1);
System.out.println(bean1.getValue()); //prints "modified"
By simply calling a method, the objects you passed to it have changed. This may not be an expected result and you have to know that is the intent of the API...i.e., you have to have carnal knowledge about it. And, do not be fooled if it has a return-type, it can still modify the reference!
Carnal Returns
Carnal returns requires significant pre-knowledge on how to handle the return.
Object o = getMyStuff();
In the above example, you have no idea what is supposed to be returned, and even worse, it may return one of, say, five different types of objects that do not have common interfaces. Although you can check/reflect (pending language) what the actual object-type is supposed to be. Horrible!!!
String result = changeThis(String rawdata);
This example is almost as bad - the returned String content may be something unexpected: i.e., could be XML, could be comma-delimited string, could be raw java/perl/php code that you are expected to run. This can be allieviated easily with documentation AND specifying in the method signature the expected result:
String result = changeThisToXML(String rawdata); //returns XML
Awareness
Just trying to share some awareness that just because you found a neat/cool way to pull something off, other people (or you using someone elses) may run into obscure or unexpected results related to Carnal Knowledge requirements. There are indeed times when you can only do it a certain way, just rememer to document and modify your method signatures to make it as clear as possible -- you never know, 5 years later you might have to use your own API/Service!
NEW: I recently learned that, surprisingly, there is functionality when writing Stored Procedures to *change* the fields in the resultset based on parameters passed in...and that people do this!! Exact same problem.
Monday, June 18, 2007
Data Improvement - Addresses
I titled this blog specifically as 'Data Improvement' instead of 'Data Assurance' or 'Data Quality'. The reason is quite simply because unless you have deterministic data coming in, you can not be assured what may be passed as data. Deterministic = there is a fixed number of values that will be accepted.
Addresses data
deterministic
A deterministic field from an address is the US State 2-digit code field. There are only 50 deterministic values acceptable, all others are rejected. These values can be cross-checked with the 5-digit zipcode (do not need a full 9-digit for State crosschecks) to ensure both the zipcode and the State code are in-sync. I like deterministic, easy to work with.
non-deterministic
A non-deterministic field is the actual address line. Attempts to improve the data on the address line include seperating the STREET physical address line and the postal MAILING address line. But, just because there are two seperate fields doesn't mean the data will be in the right place...usually when you are asking for address information, it is from a human being and human nature will kick in.
improve non-deterministic data - standards/specifications
So what can you do about these address lines? For the most part, nothing - what you get passed as data is what you have to work with. However, if you have a specific intent where you need address information to be relatively accurate, you can do something. First, determine your intent:
If you are 99% working with United States addresses and are concerned with address accuracy for actual mailings/shipments, look at some type of official CASS software. http://www.usps.com/ncsc/addressservices/certprograms/cass.htm
However, if you are trying to improve the data for the last option - individuality - and can not afford utilizing CASS software for this feature (which, btw, I highly recommend you do get CASS anyway because you can also enhance it with Address Change information), you can follow what is called 'USPS Publication 28' to standardize how the addresses look. This will not make your data foolproof by any means, but should greatly assist. Example is better:
By simply modifying the data to follow the specification, essentially 'corrected' more than 50k entries in the sample (13.75%). Now that is savings!
Addresses data
deterministic
A deterministic field from an address is the US State 2-digit code field. There are only 50 deterministic values acceptable, all others are rejected. These values can be cross-checked with the 5-digit zipcode (do not need a full 9-digit for State crosschecks) to ensure both the zipcode and the State code are in-sync. I like deterministic, easy to work with.
non-deterministic
A non-deterministic field is the actual address line. Attempts to improve the data on the address line include seperating the STREET physical address line and the postal MAILING address line. But, just because there are two seperate fields doesn't mean the data will be in the right place...usually when you are asking for address information, it is from a human being and human nature will kick in.
improve non-deterministic data - standards/specifications
So what can you do about these address lines? For the most part, nothing - what you get passed as data is what you have to work with. However, if you have a specific intent where you need address information to be relatively accurate, you can do something. First, determine your intent:
- Accurate Mailing Address
- Bulk Mailing discounts with POSTNET/barcode/zipcode sorts.
- Seperation between Street address for carrier shipment vs passing a mailing address.
- individual person identification from different data sources (i.e. john smith at 1 west rd vs 34 baltic ave).
If you are 99% working with United States addresses and are concerned with address accuracy for actual mailings/shipments, look at some type of official CASS software. http://www.usps.com/ncsc/addressservices/certprograms/cass.htm
However, if you are trying to improve the data for the last option - individuality - and can not afford utilizing CASS software for this feature (which, btw, I highly recommend you do get CASS anyway because you can also enhance it with Address Change information), you can follow what is called 'USPS Publication 28' to standardize how the addresses look. This will not make your data foolproof by any means, but should greatly assist. Example is better:
- 1 West Road vs 1 WEST RD; 1 West River Road vs 1 W RIVER RD
- P.O. Box vs PO BOX vs POBOX vs P.O.BOX
- APARTMENT # 4, APT #4, APT 4, APARTMENT 4
- 400k distinct raw address lines.
- Java-based convertor for USPS Pub 28 specification.
- 345k processed distinct address lines.
By simply modifying the data to follow the specification, essentially 'corrected' more than 50k entries in the sample (13.75%). Now that is savings!
Thursday, February 22, 2007
Useless reports and GIGO
I got pulled into a couple of meetings today. Both meetings involved doing some analysis of data submitted to me. Both results/reports had problems, per the reason for the meetings. I will talk about one of them as it is more specific to the point.
GIGO
A person (the manager) entered information through Excel. I sorted and categorized the information based on unique identifiers in one of the columns. The meeting started off with why does this report have 4 different numbers and identifiers for 'XYZ'? I should have merged together the 'X YZ', the 'XZY' (typo), and the 'X Y Z' identifiers together to get one correct number, this report is of no use, and where did I get this information?
GIGO
I tried to explain that this was the information that person entered in Excel. I then went forward to offer a couple of options on enumerated/listed identifiers so that they would be consistent. "But I do it this way because I can [enter uber entry technique process] it very fast". That is good...but why are we having this meeting? "Because the report is useless".
GIGO
The sooner you can control the input process of data, the better for everyone. I am looking for comments and suggestions to help control scenarios like this that do not involve nerf bats that have a big 'GIGO' label on it.
GIGO
A person (the manager) entered information through Excel. I sorted and categorized the information based on unique identifiers in one of the columns. The meeting started off with why does this report have 4 different numbers and identifiers for 'XYZ'? I should have merged together the 'X YZ', the 'XZY' (typo), and the 'X Y Z' identifiers together to get one correct number, this report is of no use, and where did I get this information?
GIGO
I tried to explain that this was the information that person entered in Excel. I then went forward to offer a couple of options on enumerated/listed identifiers so that they would be consistent. "But I do it this way because I can [enter uber entry technique process] it very fast". That is good...but why are we having this meeting? "Because the report is useless".
GIGO
The sooner you can control the input process of data, the better for everyone. I am looking for comments and suggestions to help control scenarios like this that do not involve nerf bats that have a big 'GIGO' label on it.
Monday, February 12, 2007
Pentaho BI
It has been several months since my last blog, and I feel like I should be struck with rosemary beads and dunked in holy water for waiting so long. But, without further ado -
Pentaho - http://www.pentaho.org - is a 'conglomerate' open source project that has put together several related projects under one Umbrella. BI, or Business Intelligence, is the entire process of obtaining, scrubbing, analyzing, reporting, and then re-analyzing/re-reporting on business data.
Pentaho has combined many of the elements to handle most of the BI stack under a friendly LGPL/no-cost license allowing you start using Business Intelligence in even the smallest of projects. That is huge...usually a project had to push over 1/4 of a million (pending your resources/randomly selected amount) before you really could engage Business Intelligence. Now, that is no longer the case. :-)
I had the wonderful opportunity to go to the Advanced Implementation Workshop in Orlando, FL from Jan 29-Feb1, 2007. Having previous projects using a portion of Pentaho, I had sufficient knowledge to get the most of the Workshop.
In addition, was able to meet with Matt Casters of the Kettle (Pentaho Data Integration) project, Julian Hyde of the Mondrian (Pentaho Data Analysis Services) project, and Thomas Morgner of the JfreeReport (Pentaho Reporting) project. This added confidence as talking with them I got a good impression they thoroughly understood their individual domains.
Pentaho is still 'in the rough' as there are occasional user-facing design and presentation items to clear up, as well as enterprise-functionality from a developer standpoint that needs to be resolved or added. Most of these are related to the full BI Suite and are usually minor (which itself is only a couple years old), while the individual projects have been around for quite some time.
Overall, I am very impressed and will start working more heavily with Pentaho, regardless if it is (initially) just reporting or more full-fledged BI solution. Now, if only they could replace jpivot...
Pentaho - http://www.pentaho.org - is a 'conglomerate' open source project that has put together several related projects under one Umbrella. BI, or Business Intelligence, is the entire process of obtaining, scrubbing, analyzing, reporting, and then re-analyzing/re-reporting on business data.
Pentaho has combined many of the elements to handle most of the BI stack under a friendly LGPL/no-cost license allowing you start using Business Intelligence in even the smallest of projects. That is huge...usually a project had to push over 1/4 of a million (pending your resources/randomly selected amount) before you really could engage Business Intelligence. Now, that is no longer the case. :-)
I had the wonderful opportunity to go to the Advanced Implementation Workshop in Orlando, FL from Jan 29-Feb1, 2007. Having previous projects using a portion of Pentaho, I had sufficient knowledge to get the most of the Workshop.
In addition, was able to meet with Matt Casters of the Kettle (Pentaho Data Integration) project, Julian Hyde of the Mondrian (Pentaho Data Analysis Services) project, and Thomas Morgner of the JfreeReport (Pentaho Reporting) project. This added confidence as talking with them I got a good impression they thoroughly understood their individual domains.
Pentaho is still 'in the rough' as there are occasional user-facing design and presentation items to clear up, as well as enterprise-functionality from a developer standpoint that needs to be resolved or added. Most of these are related to the full BI Suite and are usually minor (which itself is only a couple years old), while the individual projects have been around for quite some time.
Overall, I am very impressed and will start working more heavily with Pentaho, regardless if it is (initially) just reporting or more full-fledged BI solution. Now, if only they could replace jpivot...
Wednesday, November 08, 2006
Solve Business Problems Now, Technical Later
In working with a combination of C-level management, open source projects and developers, commercial projects and their support staff, and my own team, came to a pretty good resolution that clearly defines where I stand on certain topics:
Obviously, there is certain conditions that should be met, such as solving the business problem with some architecture/engineering to help with maintaince issues, changes, etc - but the point is to not over-engineer, and not to wait for a technical 'fix' unless it is clearly within the timeframe to solve a business problem or would impact the quality of the business solution to solve said business problem.
Cost: It *will cost more* to solve the business problem now and take care of technical issues later. This is where most management and customers do not really want to hear and, frankly, usually do not care as long as the business problem gets solved.
Revenue Stream: However, as a technical representative to an organization, and more importantly as an employee or consultant, the sooner you can bring in and/or maintain the revenue stream, the better overall for the organization. This may or may not cover the additional cost associated with the above statement, but quicker time-to-market is usually a good thing as long as the quality *of solving the business problem* is not compromised.
Why this rant? It's not a rant, it's been a thorn for a lot of individuals and teams. Some people are very good at the so-called 'quick and dirty' solutions that get something up and running, then spend *enourmous resources* maintaining that solution. Other over-engineered solutions may miss deadlines/over-budget but once deployed *may* cost much less over time (TCO) compared to an equivalent quick-and-dirty solution.
There is no perfect answer, other than no matter what a technical project will have costs during development and costs for maintanence -- but it has no value unless it solves a business problem.
- Solve the Business Problem now, if there are technical issues that will take a while to resolve, solve them later.
Obviously, there is certain conditions that should be met, such as solving the business problem with some architecture/engineering to help with maintaince issues, changes, etc - but the point is to not over-engineer, and not to wait for a technical 'fix' unless it is clearly within the timeframe to solve a business problem or would impact the quality of the business solution to solve said business problem.
Cost: It *will cost more* to solve the business problem now and take care of technical issues later. This is where most management and customers do not really want to hear and, frankly, usually do not care as long as the business problem gets solved.
Revenue Stream: However, as a technical representative to an organization, and more importantly as an employee or consultant, the sooner you can bring in and/or maintain the revenue stream, the better overall for the organization. This may or may not cover the additional cost associated with the above statement, but quicker time-to-market is usually a good thing as long as the quality *of solving the business problem* is not compromised.
Why this rant? It's not a rant, it's been a thorn for a lot of individuals and teams. Some people are very good at the so-called 'quick and dirty' solutions that get something up and running, then spend *enourmous resources* maintaining that solution. Other over-engineered solutions may miss deadlines/over-budget but once deployed *may* cost much less over time (TCO) compared to an equivalent quick-and-dirty solution.
There is no perfect answer, other than no matter what a technical project will have costs during development and costs for maintanence -- but it has no value unless it solves a business problem.
Friday, October 06, 2006
Accounting with EJB3, JPA-QL, and Databases
Just an update post on what I have been working on. I have been working on EJB3.
I started learning/using EJB3 back in Oct '05 and deployed an application based on jboss EJB3-embedded (actually, it was just the hibernate/JPA persistence side of it).
New project has allowed me to jump back into EJB3 and glad to see there is still some momentum. The persistence side (JPA) is working very well for me. The concept of using java objects through O/R mapping to deal with database structures is wonderful (and significantly improved since my days of working with EJB1.1). The new JPA-QL query enchancements over the old EJB-QL are great.
However, one problem still isn't solved completely - working with Monetary figures. I'm working on an application that needs to do financial calculations. Naturally, I'm trying to use java-side types of BigDecimal so I can set the precision/scale to only have two numbers after the decimal. On the application side, this works great. Store the data as database fieldtypes of Decimal with the same precision/scale also works great.
That's when I ran into big problems. Doing calculations on those values through a query was not working correctly. When asking the java object its value was returning values of not the correct scale.
JPA-QL statement:
Floating point optimizations - computer hardware and underlying operating systems will take shortcuts to improve the Floating Point calculation performance. These are also called single-precision Floating Point calculations. The results of using these kinds of calculations can turn something simple like '58.11' to actually be '58.10997856788934...'. Big Problem.
Solutions:
I went with the 3rd option once I understood where my problem was coming from. I was developing on Mysql 4.1 which, even though it is storing the Decimal field with the correct precision, running queries with those fields ran with the Floating Point problem. Mysql > 5.0.3 corrected this problem, as well as testing on Postgresql 8.1.1 also corrected this problem.
As for asking for a specific field value, I was still having problems. In EJB3, you can set a BigDecimal with this annoation for persistence:
As of Jboss EJB3 RC9, this only helps when saving the data to the database (i.e. the value stored in the database is correct), but not when displaying a retrieved value. This goes into how the underlying JDBC driver works with the persistence engine and how JDBC does things like 'getObject' that may return something different than expected.
Solutions:
As you can see, financial applications you need to be very careful about the Floating Point Optimization problem, and requires careful diligence across the entire stack - hardware, OS, database, and application.
6/20/2011 update: http://www.codeproject.com/Tips/50340/MSSQL-doesn-t-completely-support-IEEE754-floating-.aspx
I started learning/using EJB3 back in Oct '05 and deployed an application based on jboss EJB3-embedded (actually, it was just the hibernate/JPA persistence side of it).
New project has allowed me to jump back into EJB3 and glad to see there is still some momentum. The persistence side (JPA) is working very well for me. The concept of using java objects through O/R mapping to deal with database structures is wonderful (and significantly improved since my days of working with EJB1.1). The new JPA-QL query enchancements over the old EJB-QL are great.
However, one problem still isn't solved completely - working with Monetary figures. I'm working on an application that needs to do financial calculations. Naturally, I'm trying to use java-side types of BigDecimal so I can set the precision/scale to only have two numbers after the decimal. On the application side, this works great. Store the data as database fieldtypes of Decimal with the same precision/scale also works great.
That's when I ran into big problems. Doing calculations on those values through a query was not working correctly. When asking the java object its value was returning values of not the correct scale.
JPA-QL statement:
- "select node from Balances node where (node.totalPaid - node.tax - node.price != 0)"
Floating point optimizations - computer hardware and underlying operating systems will take shortcuts to improve the Floating Point calculation performance. These are also called single-precision Floating Point calculations. The results of using these kinds of calculations can turn something simple like '58.11' to actually be '58.10997856788934...'. Big Problem.
Solutions:
- Hardware/OS enable of double-precision floating point calculation (usually requiring Xeon-type processors). Specifically, on the database server that is running these queries (and on the application server would not hurt).
- Application-level IEEE-754 Floating Point calculations - i.e., get all the record results from the database and have the application itself do controlled calculations (in Java, strictfp keyword or use BigDecimal java.math functions).
- Carefully choose/configure your database server on how it handles Decimal-field calculations.
I went with the 3rd option once I understood where my problem was coming from. I was developing on Mysql 4.1 which, even though it is storing the Decimal field with the correct precision, running queries with those fields ran with the Floating Point problem. Mysql > 5.0.3 corrected this problem, as well as testing on Postgresql 8.1.1 also corrected this problem.
As for asking for a specific field value, I was still having problems. In EJB3, you can set a BigDecimal with this annoation for persistence:
- @javax.persistence.Column(precision=8, scale=2)
As of Jboss EJB3 RC9, this only helps when saving the data to the database (i.e. the value stored in the database is correct), but not when displaying a retrieved value. This goes into how the underlying JDBC driver works with the persistence engine and how JDBC does things like 'getObject' that may return something different than expected.
Solutions:
- Carefully choose JDBC driver/persistence engine for BigDecimal support (i.e. getObject returns double or BigDecimal, instead of float).
- Handle the returned data on your getter (re-set the BigDecimal scale and have any UI display the BigDecimal as a formatted string).
As you can see, financial applications you need to be very careful about the Floating Point Optimization problem, and requires careful diligence across the entire stack - hardware, OS, database, and application.
6/20/2011 update: http://www.codeproject.com/Tips/50340/MSSQL-doesn-t-completely-support-IEEE754-floating-.aspx
Friday, September 01, 2006
Parsing X12 EDI HIPAA 835 files
At work, I have been tasked to work on parsing and creating 835 files. I already have a previous system that parses 835 files for use in a sales/payment reconciliation program, but that uses an external commercial program to convert to XML and I process the XML results.
First, there are a number of solutions already out there. As for open-source, python-based pyx12 was one of the first I started playing with because there was a lack of non-GPL java based ones. The java-based ones include EDIReader and OBOE.
Now, one of the challenges I was running into was this whole conversion to XML. I will need to store the 835 EDI files, and if you use a different tool later for XML conversion it may convert it differently, which may lead to different results on the same 835 EDI file (or using the same original datasources when creating an 835 file).
So, I have written my own, specific parser/renderer based on a simplified, java-based domain model (domain objects that could be extended to be EJB3 Entities to be precise). No XML. And, to be more specific, it only works with Pharmacy-based 835 making it smaller and more efficient for my particular purpose. How efficient? I can parse a 2000 transaction 835 EDI file from local disk into java classes in about 200ms -- and that is prototype, non-optimized code with no external library dependencies ;-)
What have we learned? I left some out, but you have to already know and understand 835 EDI files to accomplish this. If you do not have domain knowledge related to 835's, there are plenty of software packages to help you and then you will have to integrate it into your solution. I took the latter approach first and it did work, but proved inefficient and dependent on a commercial solution. With more knowledge, you are able to take control of your destiny (with software at any rate)!
First, there are a number of solutions already out there. As for open-source, python-based pyx12 was one of the first I started playing with because there was a lack of non-GPL java based ones. The java-based ones include EDIReader and OBOE.
Now, one of the challenges I was running into was this whole conversion to XML. I will need to store the 835 EDI files, and if you use a different tool later for XML conversion it may convert it differently, which may lead to different results on the same 835 EDI file (or using the same original datasources when creating an 835 file).
So, I have written my own, specific parser/renderer based on a simplified, java-based domain model (domain objects that could be extended to be EJB3 Entities to be precise). No XML. And, to be more specific, it only works with Pharmacy-based 835 making it smaller and more efficient for my particular purpose. How efficient? I can parse a 2000 transaction 835 EDI file from local disk into java classes in about 200ms -- and that is prototype, non-optimized code with no external library dependencies ;-)
What have we learned? I left some out, but you have to already know and understand 835 EDI files to accomplish this. If you do not have domain knowledge related to 835's, there are plenty of software packages to help you and then you will have to integrate it into your solution. I took the latter approach first and it did work, but proved inefficient and dependent on a commercial solution. With more knowledge, you are able to take control of your destiny (with software at any rate)!
Friday, July 14, 2006
File Cabinets and Solutions (JCR/WEBDAV)
I'm back with my first 'official' blog in the development realm. I'm a java developer, an open-source follower, and am a big fan of consistency and standards.
My off-coding time is actually spent as a CDIA+, certified document imaging architect. One of the problems that usually fall under this role is in the arena of Document Management.
Document Management: Very simply, look at file cabinets. There are documents (a document is a collection of pages usually either single or collected together by a staple or a paper clip). Documents are stored in Folders. The folders usually have tabs that 'index' the type of documents it contains. Drawers are labeled usually with a portion of the index (i.e. if by last name, maybe the first two letters) for faster manual lookup. Everyone understands filing cabinets.
So, how come transforming the manual filing cabinet is so difficult electronically? The information and idealogy is right there in front of us!
The answer, in my opinion, is that not one solution does it the same as another - they are all proprietary and, as such, not everyone understands how to use between different proprietary document management solutions.
Solution: I have been tracking a protocol that allows you to store documents, archive documents, index documents, and then search for those documents later in a consistent fashion. This protocol is HTTP-based and is not tied to Java/.NET/Ruby/Perl, but can be used and understood by all of them. It is documented and RFC'd a number of times. What is this magical protocol? An old one called WebDAV.
I have been anxious for a solid, working solution of a java-based open-sourced WebDAV server from http://jakarta.apache.org/slide/. Don't hold your breath - after two years, I still haven't been impressed enough to actually deploy it, but am using it as a learning tool for WebDAV. Last binary was from 2004, and I have had sporadic success with building from CVS (i.e. something that used to work would break - not fun).
There is a newly active content repository from http://jackrabbit.apache.org/ is in development based on the Java Content Repository API (JCR), also known as JSR-170. I'm not overly excited about JCR/JSR-170 as it is language specific, but they are working on a WebDAV interface which I am very excited for. Come on apache, don't let me down a second time!
My off-coding time is actually spent as a CDIA+, certified document imaging architect. One of the problems that usually fall under this role is in the arena of Document Management.
Document Management: Very simply, look at file cabinets. There are documents (a document is a collection of pages usually either single or collected together by a staple or a paper clip). Documents are stored in Folders. The folders usually have tabs that 'index' the type of documents it contains. Drawers are labeled usually with a portion of the index (i.e. if by last name, maybe the first two letters) for faster manual lookup. Everyone understands filing cabinets.
So, how come transforming the manual filing cabinet is so difficult electronically? The information and idealogy is right there in front of us!
The answer, in my opinion, is that not one solution does it the same as another - they are all proprietary and, as such, not everyone understands how to use between different proprietary document management solutions.
Solution: I have been tracking a protocol that allows you to store documents, archive documents, index documents, and then search for those documents later in a consistent fashion. This protocol is HTTP-based and is not tied to Java/.NET/Ruby/Perl, but can be used and understood by all of them. It is documented and RFC'd a number of times. What is this magical protocol? An old one called WebDAV.
I have been anxious for a solid, working solution of a java-based open-sourced WebDAV server from http://jakarta.apache.org/slide/. Don't hold your breath - after two years, I still haven't been impressed enough to actually deploy it, but am using it as a learning tool for WebDAV. Last binary was from 2004, and I have had sporadic success with building from CVS (i.e. something that used to work would break - not fun).
There is a newly active content repository from http://jackrabbit.apache.org/ is in development based on the Java Content Repository API (JCR), also known as JSR-170. I'm not overly excited about JCR/JSR-170 as it is language specific, but they are working on a WebDAV interface which I am very excited for. Come on apache, don't let me down a second time!
First blog post
Welcome everyone!
I already had my first blog planned out. Unfortunately, I will be posting that blog second.
Two influential developers that I follow are Matt Raible and Rick Hightower. They use jroller.com for their blogs. I waited two months and jroller still hasn't fixed their new user registration.
Blogger.com/blogspot.com was my second choice and everytime I've come here they have been reliable and always up. Guess what, blogger.com/blogspot.com got my business :-)
I already had my first blog planned out. Unfortunately, I will be posting that blog second.
Two influential developers that I follow are Matt Raible and Rick Hightower. They use jroller.com for their blogs. I waited two months and jroller still hasn't fixed their new user registration.
Blogger.com/blogspot.com was my second choice and everytime I've come here they have been reliable and always up. Guess what, blogger.com/blogspot.com got my business :-)
Subscribe to:
Posts (Atom)