FoxPro and .NET Interop
Observation about loading executables and quick question about UNloading
Gravatar is a globally recognized avatar based on your email address. Observation about loading executables and quick question about UNloading
  Joe Kaufman
  All
  Nov 8, 2018 @ 07:58am

Rick,

From our other thread, I have been playing with using LoadAssembly() on EXE assemblies instead of DLLs. It does work fine, I can even run some console apps by invoking the Main() method. One caveat, though: The auto-assembly resolution I have built into my EXEs to find and use embedded assemblies as needed does not work. I have to pull required DLLs out and have them alongside the executable, something I do not need to do when running the executable directly. I imagine this has to do with threading, and the auto-resolution never getting triggered when bits and pieces of the assembly are called from elsewhere. So, for anyone using some sort of embedded assembly scheme to generate truly standalone applications, you'll probably want to use a scheme that plucks out the assemblies and places them as files alongside the executable (I think that is how some of the other schemes do it, anyway). Pull out all required assemblies via a method that can be called from the bridge first, and there you have it. Of course, folks who do full, setup-based deployments (or always register everything in the GAC) don't have to worry.

A related question to loaded assemblies and dependent files is one of file locks. Even after all operations have run to completion and I have performed a CLEAR DLLS, CLEAR ALL, and RELEASE ALL, I cannot delete or update loaded assemblies. I am explicitly told that VFP still has them in use. Quitting VFP entirely frees them, so that means VFP exes that run to completion and close will have no problem. But is there a better way to "unlock" assemblies that have been loaded explicitly or as needed by dependent code? It is a bit cumbersone when debugging and working on both the .NET and VFP side when one has to shut down VFP entirely to re-compile updated assemblies.

Thanks,

JoeK

Gravatar is a globally recognized avatar based on your email address. re: Observation about loading executables and quick question about UNloading
  Rick Strahl
  Joe Kaufman
  Nov 9, 2018 @ 12:30pm

What exactly are you using for the bundling?

I just tried this here on one of my ILMerge'd executables and I was able to run them fine. ILMerge makes all the assemblies available as part of the ILMerged assembly (an .exe in this case).

Which means that dependent embedded 'assembly' is now part of the exe and loads all at once.

So in theory this should just work.

+++ Rick ---

Gravatar is a globally recognized avatar based on your email address. re: Observation about loading executables and quick question about UNloading
  Joe Kaufman
  Rick Strahl
  Nov 9, 2018 @ 01:00pm

Rick,

I cannot use ILMerge because it fails when you have two assemblies loaded that have conflicting names for things. For example, (working from memory here), *DataTable *is a class name used by both System.Data and the Excel interop. Though I eventually ended up just using dynamic types for Excel COM automation, I didn't want to end up with conflicts down the road (but I agree that ILMerge is slick, and maybe it is improved now).

Instead, I add this XML to a .NET project's "csproj" file:

<Target Name="AfterResolveReferences">
    <ItemGroup>
        <EmbeddedResource Include="@(ReferenceCopyLocalPaths)" Condition="'%(ReferenceCopyLocalPaths.Extension)' == '.dll'">
            <LogicalName>%(ReferenceCopyLocalPaths.DestinationSubDirectory)%(ReferenceCopyLocalPaths.Filename)%(ReferenceCopyLocalPaths.Extension)</LogicalName>
        </EmbeddedResource>
        <EmbeddedResource Include="@(ReferenceCopyLocalPaths)" Condition="'%(ReferenceCopyLocalPaths.Extension)' == '.exe'">
            <LogicalName>%(ReferenceCopyLocalPaths.DestinationSubDirectory)%(ReferenceCopyLocalPaths.Filename)%(ReferenceCopyLocalPaths.Extension)</LogicalName>
        </EmbeddedResource>
    </ItemGroup>
</Target>

This forces the build process to pull in any assemblies that are referenced in the project (automatically, no special build scripts or parameters required).

The second part is to set an entry point for the project containing code to auto-resolve an assembly when it is called upon:

    public class AutomaticAssemblyResolution
    {
        [STAThreadAttribute]
        public static void Main()
        {
            AppDomain.CurrentDomain.AssemblyResolve += ResolveAssembly;
            DefaultEntryPoint.Main();
        }

        private static Assembly ResolveAssembly(object sender, ResolveEventArgs args)
        {
            // Find the assembly in the embedded resources for the application. To get external assemblies (DLLs & EXEs) added as embedded resources,
            // the following code can be added to the .csproj file for the application:
            //
            // <Target Name="AfterResolveReferences">
            //      <ItemGroup>
            //          <EmbeddedResource Include="@(ReferenceCopyLocalPaths)" Condition="'%(ReferenceCopyLocalPaths.Extension)' == '.dll'">
            //              <LogicalName>%(ReferenceCopyLocalPaths.DestinationSubDirectory)%(ReferenceCopyLocalPaths.Filename)%(ReferenceCopyLocalPaths.Extension)</LogicalName>
            //          </EmbeddedResource>
            //          <EmbeddedResource Include="@(ReferenceCopyLocalPaths)" Condition="'%(ReferenceCopyLocalPaths.Extension)' == '.exe'">
            //              <LogicalName>%(ReferenceCopyLocalPaths.DestinationSubDirectory)%(ReferenceCopyLocalPaths.Filename)%(ReferenceCopyLocalPaths.Extension)</LogicalName>
            //          </EmbeddedResource>
            //      </ItemGroup>
            // </Target>
            //

            // *** NOTE *** Some assemblies cannot be embedded and auto-resolved in this way.
            // For example, some DLLs needed for reporting need to be properly installed or placed
            // next to the executable in order to be found. Such run-times will be placed in a
            // global, networked location so that they can be pulled down if they are different
            // from what is locally accessible.

            Assembly executingAssembly = Assembly.GetExecutingAssembly();
            AssemblyName assemblyName = new AssemblyName(args.Name);

            string path = assemblyName.Name + ".dll";
            string[] availableResourceStreams = executingAssembly.GetManifestResourceNames()
                                                                .Select(s => s.ToUpper()).ToArray();
            if (Array.IndexOf(availableResourceStreams, path.ToUpper()) < 0)
            {
                // Assembly with "dll" extension was not found in available resources in assembly manifest. Use "exe" instead of "dll".
                path = assemblyName.Name + ".exe";
            }
            if (Array.IndexOf(availableResourceStreams, path.ToUpper()) < 0)
            {
                // Assembly with "exe" extension was not found in available resources either. Nothing to load.
                return null;
            }
            else
            {
                if (assemblyName.CultureInfo.Equals(CultureInfo.InvariantCulture) == false)
                {
                    path = String.Format(@"{0}\{1}", assemblyName.CultureInfo, path);
                }
                using (Stream stream = executingAssembly.GetManifestResourceStream(path))
                {
                    if (stream == null) { return null; }
                    byte[] assemblyRawBytes = new byte[stream.Length];
                    stream.Read(assemblyRawBytes, 0, assemblyRawBytes.Length);
                    return Assembly.Load(assemblyRawBytes);
                }
            }
        }
    }

It's SET PATH and SET PROCEDURE on steroids. 😃 Essentially, this code finds the embedded assembly and loads it as a byte array without needing to write it out as a file. As long as you then make that class the default entry point, that is what will do all your assembly resolution.

It doesn't work for some assemblies such as Microsoft.SqlServer.Types.dll and Microsoft.ReportViewer.Common.dll, and I have never figured out why. I should probably revisit ILMerge and other tools to see if they would work better for me these days (I implemented all this 3-4 years ago).

I can see why ILMerge probably works with the bridge because that merges the actual CLR code, correct? The code is always there, properly linked in the assembly. Resolving on the fly (via the above methodology) only works in the AppDomain.CurrentDomain, as that is what the resolver is bound to.

Any thoughts on how to unlock an assembly that is loaded by wwDotNetBridge? I did a lot of work in both VFP and .NET yesterday, and I must have restarted VFP about a dozen times. 😃 I finally ended up utilizing the same thing we use for network-deployed desktop apps, placing updates on the network but then having them copy to a local folder and calling LoadAssembly() against that copy. I am still curious to know if there is a better way of releasing the .NET assemblies...

Thanks, JoeK

Gravatar is a globally recognized avatar based on your email address. re: Observation about loading executables and quick question about UNloading
  Rick Strahl
  Joe Kaufman
  Nov 9, 2018 @ 01:04pm

There should be no name conflicts of class names ever in .NET. As a type name is defined by its class + it's namespace. (ie. System.Data.DataTable is not the same as Office.Interop.Excel.DataTable (or whatever)). There should be no problem keeping these types seperate because the name is unique based on the namespace. The only time you run into problems if you import the same type multiple times from different assemblies. But ILMerge should be able to resolve this the same way the runtime resolves this when you run an application.

+++ Rick ---

Gravatar is a globally recognized avatar based on your email address. re: Observation about loading executables and quick question about UNloading
  Joe Kaufman
  Rick Strahl
  Nov 9, 2018 @ 01:21pm

All I can tell you is that as of 3-4 years ago -- it didn't. ILMerge would throw errors, and my research indicated it was not something easily resolved. And I am pretty sure it had to do with the Excel Interop assembly.

Hm, some haze is clearing... I think the other issue was that (as you state), it was trying to merge the same module twice. Here's why: the main assembly I load into all projects is one called "BellGlobal", because I work at a company named "Bell", and these are global routines that I want built in. I don't want to have to copy the global DLL all over the place when I make a change (we can talk pros and cons of modularity another time. 😃 ). We are deploying desktop apps, so it is not like I am just deploying to one server that serves up web services.

So, an assembly such as the Excel interop was built into BellGlobal, but also needed to be referenced in specific projects where I wanted to use it. Bam, ILMerge tries to load it twice into any such project that wants to use *BellGlobal *and the Excel interop. That kind of thing happens quickly when you start wanting to merge all assemblies into one "standalone" executable.

I spent several days making my head hurt trying to be a better system architect (my kung fu is weak) and better at segmenting functionality and modules -- to no avail. That's when I discovered the alternate solution that ended up working better for me.

These days I read a lot about Costura.Fody on threads dealing with embedding assemblies. Not sure what methodology it uses...

Thanks, JoeK

Gravatar is a globally recognized avatar based on your email address. re: Observation about loading executables and quick question about UNloading
  Rick Strahl
  Joe Kaufman
  Nov 9, 2018 @ 05:27pm

In general it's a bad idea building monolithic EXEs that are self contained. There's no benefit to that other that you can say "here's a single file, instead of 20 assemblies". The file won't be a any smaller, and it won't prevent any runtime load issues because in essence there's no difference whether files load from disk or from within the EXE with some custom runtime assembly resolver.

The .NET Runtime is very smart about resolving dependency issues with the same type loaded. Pre-compiling tools can't be as smart because they don't have the same information that .NET has at runtime.

IAC - I think this approach is just asking for issues down the line. I wouldn't recommend it.

+++ Rick ---

Gravatar is a globally recognized avatar based on your email address. re: Observation about loading executables and quick question about UNloading
  Joe Kaufman
  Rick Strahl
  Nov 9, 2018 @ 07:13pm

Rick,

I'm not sure it is as simple as pooh-pooing "here's a single file, instead of 20 assemblies". We are a desktop house. How do you deploy 20 assemblies (and have them be replaceable on the fly) in a Windows network environment? I'm not being rhetorical -- I'm all ears.

So, no, I won't just give up on an easy network deployment, having a single file that is easy to copy locally and run. I can call a user at the busiest moment of the day and say, "Exit the app and get back in. Cheers!" In a non web-based environment, what is the best-case idea on employing rapid, dynamic updates (without spending a lot of money on some sort of application delivery framework)? I don't currently have the clout to tell my company they need to change their entire deployment paradigm, and we have seen no downside doing what we have been doing (for about 15 years, starting with Foxpro). That's hard to argue with. That's why I copied the design from VFP to .NET. It isn't broken, so I don't see why I should try to fix it.

After all, aren't you the guy who designed a way to swallow .NET errors? 😃 Which I think is completely brilliant, by the way.

Thanks,

JoeK

Gravatar is a globally recognized avatar based on your email address. re: Observation about loading executables and quick question about UNloading
  Rick Strahl
  Joe Kaufman
  Nov 9, 2018 @ 07:31pm

Whether you copy one file or 20 files in a flat folder hierarchy makes no difference other than copying 20 files vs 1. Is that really a problem? In either case you have to exit the app and make sure the app is unloaded whether you replace one file or 20. If you're doing on the fly updates the files can be zipped and unpacked as part of the app update.

Especially if you are automating this process with some sort of automated process or even just a batch/ps file.

I just don't see any benefit to the single file other than optics. I'd like to understand what beyond optics (and single file copy vs. 20 file copy) is the benefit?

That's hard to argue with, isn't it? That's why I copied the design from VFP to .NET. It isn't broken, so I don't see why I would need to fix it.

Sure, if it works it works and that's great. Just seems there's extra effort involved in that solution (packaging) with potential issues for other things. ie. trying to get at your assemblies which started this whole thread originally. That likely isn't working because of your bundling. That's side effects...

After all, aren't you the guy who designed a way to swallow .NET errors? ?? Which I think is completely brilliant, by the way.

LOL - right. I guess that's a matter of opinion and style.

wwDotnetBridge doesn't swallow errors. It catches them and passes them on explicitly - by default. If you want exceptions that are passed thrgouh, you can turn them on.

I treat errors on these objects like I would a service - you should explicitly check for errors rather than then let them ripple up where you're going to have a real hard time figuring out what to do with them because they are nasty COM errors that you can't pass into the UI (or most other things) without a bunch of post processing.

+++ Rick ---

Gravatar is a globally recognized avatar based on your email address. re: Observation about loading executables and quick question about UNloading
  Joe Kaufman
  Rick Strahl
  Nov 10, 2018 @ 06:35am

Yeah, wwDotNetBridge handles errors perfectly. I was surprised how it just passed everything through without me needing to do a thing (and I put it through the wringer of error scenarios, both intentionally and, er, not).

It is true that our "copy to local" routine could just as easily check 20 files as 1, I just didn't want to re-write our VFP and C# launchers to do that. Moving one file can practically be hard-coded. But in the 20-file scenario, it is invariably going to eventually go to 21, then 22, then 30, etc. I would need to change config files or meta-data somewhere for all apps if I deployed a new, global module (we have around 15 applications that tap into the global functionality). We don't have that infrastructure set up. Again, we could do it, and it would likely be the "right" way, I just decided to take on that complexity at build time (where it was easier to automate, IMO) rather than via some more complex config framework. A monolithic EXE isn't all that different from a library of assemblies in a ZIP file.

The truly RIGHT way to do all this, of course, is to have everything on a server tier. Then build and deployment are more one-in-the-same. At this point we are de facto moving in that direction because we are looking into some sort of ERP platform for the company. I imagine the bridge is going to be VERY useful during conversion and transition to that new platform, as many ERP solutions use .NET as their foundation (we are sticking with Windows). You have made it possible for me to have one foot in VFP and the other in .NET, and it has worked well beyond my hopes that it would.

Thanks,

JoeK

Gravatar is a globally recognized avatar based on your email address. re: Observation about loading executables and quick question about UNloading
  Rick Strahl
  Joe Kaufman
  Nov 11, 2018 @ 04:18pm

Yup - I don't think we disagree on anything - things get built with one thing in mind and then things change and the end of the day solution isn't always the best way, but it works and then you have lots of inertia to change. I know how that goes, trust me...

I wasn't trying to criticize your set up BTW, I was merely trying to point out that IMHO there's little benefit to packaging. Clearly your solution works for you and there's nothing wrong with it. But in the future you might consider whether that's necessary to a deployment process. At the end of the day any automated deployment process can easily handle getting the right files copied.

You have made it possible for me to have one foot in VFP and the other in .NET, and it has worked well beyond my hopes that it would.

Thank you - I've heard that from a lot of people and that's awesome. It's certainly true for me. I don't have a lot of code left in FoxPro, but the things that do all heavily interop with .NET and wwDotnetBridge has made that a breeze.

Thanks - good discussion and it's resulted in a couple of small improvements in wwDotnetBridge (Generics access and also a properly documented property to turn on the exception passthrough (it was there before but not documented and in a buried method)).

+++ Rick ---

Gravatar is a globally recognized avatar based on your email address. re: Observation about loading executables and quick question about UNloading
  Joe Kaufman
  Rick Strahl
  Nov 12, 2018 @ 08:26am

Rick,

If my side of the discussion helped wwDotNetBridge become even better, then that is great (but you did all the work!). 😃

Oh, back to the original question before I forget -- unloading assemblies to prevent lock issues? Is there any facility for that, or am I missing a CLEAR or RELEASE command in VFP that will help? If there is a best-practice for "closing" the bridge when done, I'd like to follow it...

Thanks,

JoeK

Gravatar is a globally recognized avatar based on your email address. re: Observation about loading executables and quick question about UNloading
  Rick Strahl
  Joe Kaufman
  Nov 12, 2018 @ 12:16pm

Assemblies cannot be unloaded from an AppDomain. Once loaded they stay loaded.

The only way to get around this is to create a new AppDomain then load assemblies inside of that, then release the AppDomain. It's doable (and what I do in Help Builder for loading addins) but it's crazy ugly and difficult to pass data around because data has to be serialized from the main appdomain to the separate app domain and you're bound by serialization rules (MarshalByRef objects required or [Serializable]).

Bottom line - you should assume you can't unload assemblies.

+++ Rick ---

Gravatar is a globally recognized avatar based on your email address. re: Observation about loading executables and quick question about UNloading
  Joe Kaufman
  Rick Strahl
  Nov 12, 2018 @ 12:25pm

Rick,

OK, I'll certainly trust you on the complexities, and it is a small price to pay for all the doors this opens.

Thanks again,

JoeK

© 1996-2018