Web Connection
Modernizing Antique WW site: from 3.x to 6.x and beyond
Gravatar is a globally recognized avatar based on your email address. Modernizing Antique WW site: from 3.x to 6.x and beyond
  John R
  All
  Oct 22, 2019 @ 03:24pm

This year we had reason to modernize a very old WW app.

We'd started to run into difficulties with Ghostscript privileges and customers struggling with manual installation with numerous tweaks to make things work on newer IIS and Servers. Latest WebConnect features like the CONFIG parameter that automates almost all configuration, were too good to pass up.

I thought I'd create this thread to document procedures followed, to minimize pain for anybody else considering restoration of their antique!

Initial review confirmed several areas needing address, including Script Handling, Session handling, and use of profoundly obsolete classes needing to be modernized.

I started with the WW Scripts. The decision had been made years ago only to use compiled scripts included in the project. Rather than the recompilation offered in admin.asp (now admin/admin.aspx) a compilescripts.prg had been written to "compile" all .htm files in a folder, to prgs that can be included in the project. This was updated to use wwScript's ConvertASPtoVFPcode function that can accept a source string and return the WW compiled result. ConvertASPtoVFPcode's use of string input and output makes it easy to reliably tweak source or output for better result, rather than editing the original source that then needs a diligent source control in case something done 3 edits ago is discovered to create havoc.

Immediate observation:

  • ConvertASPtoVFPcode replaces the original TEXT...ENDTEXT with a series ofResponse.Write() commands.

While TEXT...ENDTEXT can handle expressions of any type, Response.Write works with strings- meaning ConvertASPtoVFPcode needs to ensure everything it puts in Response.write is a string, or is made into a string. Usually this would require parsing and evaluating at compile time to handle conversion of non-string types- but ConvertASPtoVFPcode achieves it by wrapping all the script's output in VFP's TRANSFORM(EVALUATE([expression goes here])) wrappers.

Which creates the next point:

  • Because output is wrapped in [ ], extra processing is required if expressions, javasacript or HTML themselves contain [ ] ; and
  • Since string literals can't exceed 255 bytes, large chunks of HTML or expressions may need to be split and concatenated inside the EVALUATE().

ConvertASPtoVFPcode is pretty good at spotting and breaking up oversized string literals and handling [ ] in your VFP expressions or HTML- but this adds complexity and no doubt a performance hit that could have been avoided if only scripts used " or ' rather than [ ] and always use string expressions in <%= %> - more on that later.

In the early 2000s, use of [ ] was pretty popular and most of the scripts I wanted to modernize used [ ] widely. While ConvertASPtoVFPcode usually handles square brackets by wrapping them in a string concatenation, in 2 scripts I hit issues with convoluted structure somehow ending up with an oversized string literal provoking an error when the script is run. I don't regard this as a bug; it's down to complexity and it's pretty impressive that ConvertASPtoVFPcode gets it right almost all the time in our hundreds of scripts.

I decided to fix it by replacing [ ] where possible. Since ConvertASPtoVFPcode accepts input strings, I decided to write a preprocess routine that would munge the source string.

This led to discovery that [ ] also are widely used for array brackets (myarray[1,2]) and in Javascript so that initial conversion to " or ' created plenty of errors and proved the wisdom of preprocessing rather than editing the source! In our scripts it turned out that use of [ ] in Javascript was limited to only a few areas, ditto arrays where only 2 array names were used with [ ]. This could be handled relatively easily as follows.

*---use of laFields[x,y] messes up conversion
liAt=ATC("lafields[",m.lcSource)
DO WHILE m.liAt>0
	lcSource=LEFT(m.lcSource,m.liAt+7)+"("+STRTRAN(SUBSTR(m.lcsource,m.liAt+9),"]",")",1,1)
	liAt=ATC("lafields[",m.lcSource)
ENDDO
	
liAt=ATC("laVars[",m.lcSource)
DO WHILE m.liAt>0
	lcSource=LEFT(m.lcSource,m.liAt+5)+"("+STRTRAN(SUBSTR(m.lcsource,m.liAt+7),"]",")",1,1)
	liAt=ATC("laVars[",m.lcSource)
enddo	 

*---goodbye, square brackets!
licnt=1
liat=AT("[",m.lcsource,m.licnt)
DO WHILE m.liat>0
	lcleft=LEFT(m.lcsource,m.liat-1)
	lcright=SUBSTR(m.lcsource,m.liat+1)
	*---leave javascript alone
	IF LOWER(RIGHT(m.lcLeft,6))="frames" OR LOWER(RIGHT(m.lcleft,6))="_nodes" OR LOWER(RIGHT(m.lcLeft,3))=" me" ;
		OR LOWER(RIGHT(m.lcLeft,5))=".rows" OR RIGHT(m.lcLeft,2)=[,"] OR RIGHT(m.lcLeft,1)=["] and INLIST(LOWER(m.lcright),"system"," ]","x]")	
		licnt=m.licnt+1
		liat=AT("[",m.lcsource,m.licnt)
		LOOP
	endif
			
	liat2=AT("]",m.lcright)
	lctest=LEFT(m.lcright,m.liat2-1)
	DO case
	
	CASE ["] $ m.lctest
		IF ['] $ m.lctest
			licnt=m.licnt+1
			liat=AT("[",m.lcsource,m.licnt)
			loop
		ELSE
			lcconv=[']
		ENDIF
	otherwise
		lcconv=["]
	ENDCASE
			
	lcsource=m.lcLeft+m.lcConv+STRTRAN(m.lcright,"]",m.lcConv,1,1)
	liat=AT("[",m.lcsource,m.licnt)
enddo

This resulted in much cleaner compiled scripts with no oversized literals or other faults.

One last thing to watch: use of [ ] as string delimiter can cause unexpected substitutions. Try this:

#DEFINE DINNER go away! 
? [they invited me to dinner]

With all script HTML and expressions now wrapped in [ ], this is something new you might encounter.

More to follow...

Gravatar is a globally recognized avatar based on your email address. re: Modernizing Antique WW site: from 3.x to 6.x and beyond
  Rick Strahl
  John R
  Oct 22, 2019 @ 07:51pm

Thanks John,

Just for your information: ConvertAspToVfp() at runtime only runs once to compile the page - once it's compiled it's reusing the compiled code, so the overhead of parsing isn't much of an issue for performance.

The code has see a lot of edge cases. Converting code dynamically into strings is difficult due to parsing characters and parse sequences. FoxPro is not really good at string parsing and tokenizing, so it's tough to build a very high quality code parser. If you do run into specific issues please post them though and see if we can find a workaround that doesn't break something else 😃

Overall though I think this is pretty close to as good as it's going to get.

The #DEFINE issues with [ shouldn't be a big problem since we're talking about script code. I don't see much use for pulling in #DEFINE statements or even declaring them in single file script components. In those cases it's probably just as easy to declare a variable.

Thanks for the feedback - do appreciate it.

Gravatar is a globally recognized avatar based on your email address. re: Modernizing Antique WW site: from 3.x to 6.x and beyond
  John R
  Rick Strahl
  Oct 23, 2019 @ 12:44pm

Just for your information: ConvertAspToVfp() at runtime only runs once to compile the page - once it's compiled it's reusing the compiled code, so the overhead of parsing isn't much of an issue for performance.

Got that - but string concatenation to avoid oversize literals and to wrap [ and ] characters must add overhead at runtine. Having determined that widely used [ ] is gratuitous most of the time when " or ' could be used, I decided it's a better fit with your new methodology to get rid of [ ] where possible. There were a few locations where I couldn't, mostly in embedded Javascript using [ ] or concatenation like lcJavascript=m.lcJavascript+[confirm("John's message already uses both sorts of bracket")]

One other thing on this topic: if not for concatenation for [ ] and if you could be sure that every expression in the script is a string, theoretically it might not need the TRANSFORM(EVAL()). I did some experimentation so that if the response.write parameter doesn't contain +"["+ or +']'+ and if it contains TRIM( or STR( or LEFT( RIGHT( STUFF( etc etc... maybe even TRANSFORM() with only one parameter - then I might get away with no TRANSFORM(EVAL()) . I have not done performance testing to see whether this level of effort is worth it.

Reference to wc.dll

Next, you'll want to convert any wc.dll references to use scriptmaps and the new NET rather than the old ISAPI extensions. I found plenty of URLs that included wc.dll rather than scriptmaps. Rick has been advocating scriptmaps rather than wc.dll for most of this century, which confirms just how old some of this stuff is!

If you also have scripts containing wc.dll urls, I won't belabor the ease with which VFP can STRTRAN() wc.dll to your new script map. In fact you could create a *.dll scriptmap and leave wc.dll references as is - but I was concerned at the risk of unpredictable behavior if somebody ever managed to leave wc.dll in place with a few steps not completed during upgrade. Better to eliminate completely.

The good news is that modern IIS makes it terrifically easy to apply script maps - by editing web.config in your new web folder. In the following example, a text editor is used to add the first Handler to those automatically applied by Rick's wizard, so that urls pointing to anything.mymap now provide WW functionality.

   <handlers>
      <add name=".mymap_wconnect-module" path="*.mymap" verb="*" type="Westwind.WebConnection.WebConnectionHandler,WebConnectionModule" preCondition="integratedMode" />
      <add name=".wc_wconnect-module" path="*.wc" verb="*" type="Westwind.WebConnection.WebConnectionHandler,WebConnectionModule" preCondition="integratedMode" />
      <add name=".wcs_wconnect-module" path="*.wcs" verb="*" type="Westwind.WebConnection.WebConnectionHandler,WebConnectionModule" preCondition="integratedMode" />
      <add name=".wcsx_wconnect-module" path="*.wcsx" verb="*" type="Westwind.WebConnection.WebConnectionHandler,WebConnectionModule" preCondition="integratedMode" />
     </handlers> 

Finally: use of scriptmap theoretically can replace one of your URL parameters, so wc.dll?myparameter1~myparameter2 can become myparameter1.mymap?myparameter2. This is fairly easy with Rick's new classes, but the scripts I'm converting have multiple variants that would require some clever parsing to automate the change. I deferred this step to allow thorough testing first.

Appmain.prg

Next would be converting the antique appmain.prg . This was less effort than expected:

Changes to Server Class Definition

The old Server Class Definition had SetServerEnvironment() and SetServerProperties() ; current WW replaces these with OnInit(), OnInitCompleted() and Onload(). There's good self-documentation in the WWskeleton created by the latest WW Wizard so I won't repeat here - but it was not difficult to split the original functions into their replacements. It's worth thinking carefully about exactly when you want things to be invoked during startup, as this new separation can make it much easier to debug and also maximize startup performance.

More follows...

Gravatar is a globally recognized avatar based on your email address. re: Modernizing Antique WW site: from 3.x to 6.x and beyond
  Rick Strahl
  John R
  Oct 23, 2019 @ 03:23pm

String concatenation in FoxPro is actually very, very fast if you do it against a local in scope memory variables. That was a scenario Calvin optimized heavily by hand in VFP precisely for Web scenarios that he and I had discussed on several occasions. Basically string concats are buffered in pre-allocated memory blocks that pre-size and are filled with data so there are relatively few (slow) memory allocations. String concats are much, much faster than running equivalent code in TEXTMERGE blocks. I can't remember the exact details but TextMerge actually dumps to intermediary file (or stream) and that overhead more than breaks the bank of performance.

String concat is about as fast as you're going to get with FoxPro output IMHO.

The issue of complexity representing 100% of FoxPro code/script as literal strings is an issue though, although as you've mentioned on your large code base, and in my experience seeing many diverse apps, it is pretty minor. Most things that do break are usually due to overly long single lines or even more rarely in nested string and square bracket scenarios which often have workarounds (once you find them which is not so nice).

+++ Rick ---

© 1996-2019