Late template instantiation

I read the TI wiki article on C++ template instantiation issues, and though my situation is different than the main problem depicted, I'm wondering if I could use one of the tricks therein it to speed up my compile times.

My code is something like this:

// File Base.hpp

template<class T>
struct Base
{
     virtual ~Base() {}
     virtual void Fn( const T& ) = 0;
};

// File Derived1.hpp

#include "Base.hpp"
struct Derived1 : Base<int>
{
     virtual void Fn( const T& );
};

// File Derived1.cpp

#include "Derived1.hpp"
void Derived1::Fn( const T& t )
{
     // ...

}

// File Derived2.hpp

#include "Base.hpp"
struct Derived2 : Base<double>
{
     virtual void Fn( const T& );
};

// File Derived2.cpp

#include "Derived2.hpp"
void Derived2::Fn( const T& t )
{
     // ...

}

// File: Main.cpp

#include "Derived1.hpp"
#include "Derived2.hpp"

int main()
{
     Derived1 d1;
     Derived2 d2;
     d1.Fn( 42 );
     d2.Fn( 4.2 );
     return 0;
}

The difference here is that I'm linking all these cpp files together in one executable, not linking to a separate lib file, but the link times for my program are quite long, which I think may be a result of late template instantiations, which triggers re-compilation of the cpp files at link time. Would you expect adding explicit instantiations along the lines of: 

template class Base<int>;
template class Base<double>;

to be of any help here? I tried adding each in Derived1.cpp and Derived2.cpp respectively, but then the linker gives duplicate symbol errors on the vtable. I can add them to Main.cpp and it links 3 minutes faster than without. Why can't I put the statements in the more logical location of the Derived?.cpp files? (In my real code, I have three derived classes, and only one of them produces a linker error when the explicit instantiations are put in the derived classes .cpp files.)

Is it possible that this could actually slow the total time for one or more files down depending on, say, the link order? For instance, say the explicit instantiations are in Derived1.cpp and Derived2.cpp, but the order of the files given to the linker is Derived1.obj, Main.obj, then Derived2.obj. Since the linker hasn't seen an instantiation of the Base<double> yet when it gets to Main.obj, does it recompile and instantiate it in Main.obj even though I have forced it to be instantiated at compile-time in Derived2.cpp? Does this mean that Base<int> was compiled/instantiated only once whereas Base<double> is instantiated twice, with one instantiation being thrown away at link time (or perhaps it isn't thrown away, and that is why I get the vtable linker error)?

I'm using CGTools 5.1.12. What is the best way to speed up building of template-laden code? Is there any way to prevent recompilation and late (possibly redundant? see above) template instantiation?