ant - download files using get

Document Actions

this is about the idea to avoid the problems with none cached downloads using the get task.

Why I'm doing this

during the last week I changed my complete buildstructure from storing my libs local to load them from the web. so I can get sure to use the newest libs and always know where I can find the needed license files.

What exist to download files from ant

ant get task
dependency task
several more, but didn't look at them

so at first I worked with the get task, but pretty far i saw, damn I have so much traffic with this, can't be. So I used the dependency task. Pretty cool, except for one thing: it works on a defined structure <example> to get the version numbers. And there are always one or two packages missing or outdated.

in my case:

eclipse was only existing at 2.x, download size ca. 60 MB
jama did'nt exist at all, download size couple of kilo bytes

so I stucked with get task for a day. The problem is. My build management system is configured for integrations taks and nightly build. So a simple example 25 Builds x 60 MB = 1,5 GB not neccessary web traffic. I didn't think about this at first, because we have a damn fast connection, so it doesn't matter. Until my connection to the eclipse website where limited when I worked at home. And it's also not the based idea to support opensource systems with wasting their bandwidth... Sorry!

After all I decided ok, I like the dependency task features, specially the caching. So lets write an own get ant task. For this i choosed the lazy way. just downloaded the existing ant source version, and copy the needed task to my project and modified it. I love opensource, makes live easier.

Result

this is the final result what does what I want.

loads files from the internet
save an actual copy at the localharddisk and use this the next time if the file is requested.
in case there server timestamp is updated we replace the cached file

Code

here is the code, it's just a fragment which shows the important part. Feel free to contact me!

apache ant license
apache ant download

////////////////////////////////////////////////////////////////////////////////////////////////////////////

    public boolean doGet(int logLevel, DownloadProgress progress) throws IOException {

        cache = new File(this.getProject().getProperty("user.home") + "/.fetch-cache");

        // dont do any progress, unless asked
        if (progress == null) {
            progress = new NullProgress();
        }

        log("Getting: " + source, logLevel);
        log("To: " + dest.getAbsolutePath(), logLevel);

        // set the timestamp to the file date.
        long timestamp = 0;

        boolean hasTimestamp = false;
        if (useTimestamp && dest.exists()) {
            timestamp = dest.lastModified();
            if (verbose) {
                Date t = new Date(timestamp);
                log("local file date : " + t.toString(), logLevel);
            }
            hasTimestamp = true;
        }

        // set up the URL connection
        URLConnection connection = source.openConnection();

        if(checkTimeStamp(timestamp,connection)) {
            return false;
        }


        InputStream is = null;

        boolean found = false;

        File cached = null;
        if (caching) {
            if (cache.exists() == false) {
                cache.mkdirs();
            }
            cached = new File(cache.getAbsolutePath() + "/" + dest.getName());

            log("Check cache for: " + cached, logLevel);

            if (cached.exists()) {
                log("found in cache", logLevel);

                //check cached timestamp

                long cacheTimestamp = cached.lastModified();

                boolean ok = checkTimeStamp(cacheTimestamp,connection);
                if(ok) {
                    log("timestamp is ok -> use cache", logLevel);

                    is = new BufferedInputStream(new FileInputStream(cached));
                    found = true;
                }
                else {
                    log("timestamp is not ok -> don't use cache", logLevel);
                    found = false;
                }

            }
            else {
                log("cache is empty", logLevel);
            }
        }
        if (found == false) {

            for (int i = 0; i < 3; i++) {
                try {
                    is = connection.getInputStream();
                    break;
                } catch (IOException ex) {
                    log("Error opening connection " + ex, logLevel);
                }
            }
            if (is == null) {
                log("Can't get " + source + " to " + dest, logLevel);
                if (ignoreErrors) {
                    return false;
                }
                throw new BuildException("Can't get " + source + " to " + dest, getLocation());
            }
        }
        FileOutputStream fos = new FileOutputStream(dest);
        FileOutputStream cacheStream = null;

        if (found == false) {
            cacheStream = new FileOutputStream(cached);
        }
        progress.beginDownload();
        boolean finished = false;
        try {
            byte[] buffer = new byte[100 * 1024];
            int length;
            while ((length = is.read(buffer)) >= 0) {
                fos.write(buffer, 0, length);

                if (found == false) {
                    cacheStream.write(buffer, 0, length);
                }
                progress.onTick();
            }
            finished = true;
        } finally {
            FileUtils.close(fos);
            FileUtils.close(is);
            if (found == false) {
                FileUtils.close(cacheStream);
            }
            // we have started to (over)write dest, but failed.
            // Try to delete the garbage we'd otherwise leave
            // behind.
            if (!finished) {
                dest.delete();
                if (found == false) {
                    cache.delete();
                }
            }
        }

        progress.endDownload();

        if (useTimestamp) {
            long remoteTimestamp = connection.getLastModified();
            if (verbose) {
                Date t = new Date(remoteTimestamp);
                log("last modified = " + t.toString() + ((remoteTimestamp == 0) ? " - using current time instead" : ""), logLevel);
            }
            if (remoteTimestamp != 0) {
                FILE_UTILS.setFileLastModified(dest, remoteTimestamp);
            }
        }

        // successful download
        return true;
    }

Created by zwluxx
Last modified 2005-10-08 04:41 PM

Metabolomics Fiehn Lab

Sections

Personal tools

Navigation

News

ant - download files using get

Document Actions