IT Archive Main

WebSpider Public Alpha Updates

This page will contain all public updates and release notes for WebSpider project.

UPDATE:Here's build 25 - There have been many substantial changes since Build 21, including a new UI, greater compliance with the HTTP standard, improved performance, (hopefully) improved stability, a few new features, and a partially redesigned core.
BUG REPORTS: Please mail Jeremy Cattone with bug reports, questions, or any other correspondence concering the WebSpider project.

If reporting bugs or enhancement requests, please include the following:

Your name
preferred email address / phone #
Description of bug / enhancement
Operating system on which the bug was found
Is your machine a single or multi-processor system
What site caused the error, or the starting site, and apx. time until the error appears
Your connection speed
The settings used during the search
If possible, please attach a copy of "autoback.sav" from the directory WebSpider was run - it is a periodic 10-minute autosave of your search that may allow for easier replication of the bug
Any additional information you feel may be helpful in diagnosing the problem

Projected Milestones (Currently Alpha Stage):

(currently apx. 80% complete, pending API / UI stabilization):

Alpha series: (In Progress)	Basic feature development and initial defect identification / characterization Initial user-based feature request generation
Beta 1:	Basic HTTP/1.1 support including authentication and alternate port specifications (does not include cookies) The ability to save and restore searches A stabilized UI including basic activity monitoring and the ability to customize basic operational parameters of the spider A stabilized API and internal design sufficiently flexible to accomodate the inclusion of all proposed Version 1 features
Beta 2:	Fundamental client / server functionality Extended ability to customize the operation of the spider The ability to edit / manipulate the search queues at runtime Content analysis and a (more) intelligent search algorithm incorporating non-adaptive keyword weighting to automatically refine searches
Beta 3:	Adaptive keyword weighting Duplicate file detection via hash scan / binary compare Extended client / server functionality Cookie / basic form-driven authentication support
Version 1:	The elimination of all outstanding functional bugs The incorporation of all appropriate feature requests for a v.1 product Stable operation on Win95 through Win2K Help files and documentation Fully stabilized and documented API
Version 2:	FTP support Support for plug-in modules that can verify the integrity of downloaded files (i.e. is able to decode image files and identify corruption) Port scanning Route tracing Site mirroring Support for scripted authentication requests

Critical bug report!
It has been reported and confirmed that the "Deskmate" animated entertainment applications cause instability and crashes in both MS Dev Studio, and in WebSpider. I suspect that there is a shared DLL that is being locked by the Deskmate applications that causes the crash, but I have yet to isolate the conditions. More info available as I resolve the issue.

Version Info:

Alpha Build 20 (10/11/00)

Not the latest build, but the latest that still compiles - will try to post a version w/ the latest mods tomorrow.

Alpha Build 17 (06/10/00, 2:30 PM)

Fixed crash condition resulting from fat (> 1024 char) URLs.

Alpha Build 16 (05/23/00, 9:00 PM)

Added bandwidth requirement restrictions - it's still a little flaky, but it works in most situations.
Performance improved slightly during long searches by eliminating several redundant checks for duplicate search targets
Improved response time while loading saved searches

Alpha Build 15 (05/13/00, 5:00 PM)

Fixed several small bugs involving the generated index file
Fixed potential crash (access violation) in the save routines
Rewrote the window snapping routines to more closely emulate WinAmp (c) style window snapping
Performance monitor now reflects bandwidth utilization more accurately, and scales more gently after a large peak passes offscreen
Fixed bug where files that were too small were being resubmitted for download multiple times

Alpha Build 14 (05/30/00, 11:45 PM)

Added the ability to save / restore searches. Searches save the history, download, and search queues, number of downloads retrieved, and target file types. Be aware that saving / loading long-running searches may take a considerable amount of time, as the save files generated may easily be several MB in size.
Most major options are now saved upon exit
Added Save on exit prompt
Disabled performance management by default as an experiment. I think this will change, but for the time being, make sure to enable performance management if you don't want the spider to continue searching, regardless of the download backlog
Enabled check options on a number of items in the Advanced menu
Known bug: If full window drag is not enabled, window snapping is not functional

Alpha Build 13 (05/29/00, 4:00 PM)

Found (finally!) and fixed the primary source of file corruption - you should see a significant increase in the integrity of downloaded files.
WebSpider now attempts to prevent the remote server from using persistent connections
Added index files for downloads

Alpha Build 12 (05/29/00, 12:00 PM)

Made clipboard capturing optional (and disabled by default.) It is now accessible via the Advanced menu
DL Throughput draws on startup now. Seems like a recurring bug.
Added per-site hash prefixes on all downloads and a menu item to enable / disable it under 'Advanced'
Fixed several minor message handling bugs

Alpha Build 11a (05/29/00, 9:30 AM)

Fixed zombie download issue where a timed out sockets could eventually force the engine to pause waiting for the timed out downloads to complete

Alpha Build 11 (05/29/00, 1:30 AM)

CRITICAL BUG: Some files are still getting corrupted due to problem in http header parser
Fixed a different file corruption and premature download abortion bug,
API: Added crude support for closing threads gracefully by passing bTerminationRequested through GetHTTPDocument. a true event would be better so we can handle the event in proper style, but at least the crashign bugs are gone.
Added graceful shutdown support for all threads. This may result in a slight delay whiile closing, but this fixes a small global resource leak that may have occurred on some systems
API: made Stop() public and moved all shutdown code from the class destructor into Stop() since the destructor will not be called before the owner window is destroyed when a CSpider object is declared as a static variable
Fixed (i think?) crash on exit due to zombie threads

Alpha Build 10a (release build only) (05/28/00, 10:20 PM)

Fixed crash in release binary when trying to write to an unopened debug log

Alpha Build 10 (05/28/00, 9:45 PM)

Fixed crash upon running out of space on the destination drive. Files will still be discarded w/ no warning, but the resulting crash condition was removed.
Several misc. typos fixed
Stop button now produces status message to indicate the stopped status
Increased snap distance to 30 pixels (from 15)
Potential deadlock on restarting / stopping the engine eliminated
Engine no longer processes queues before a start signal is received
Fixed logic error in http parser that was causing some URLs to be ignored - some still are, but no longer from HREF or SRC tags
Bug where URLs crossing 8K boundaries were not being recognized is now fixed
Improved response time during stop / restart / quit, I suspect the fix is causing crashes on exit though.

Alpha Build 9 (05/27/00, 8:00 PM)

Bandwidth monitor is now active at startup
Fixed condition where a 404 error could prevent an attempt at retreiving the document via an alternate URL

Alpha Build 8 (05/24/00, 3:15 PM)

The 'Play' button now defaults to the correct state (Play, engine stopped)
The 'Play' button only needs to be pressed once after stopping / pausing a search
Added rudimentary clip-board capture for URLs (on by default, cannot turn it off yet)
Throughput monitor now redraws even when engine is stopped
Added memory constraint option
Stopping the engine no longer requires a program restart to restart the search engine

Alpha Build 7 (05/24/00, 2:15 AM: Initial update)

Unknown hosts no longer produce huge numbers of message boxes
Version information has been made consistent
Infinite loop on DNS resolution errors fixed

Alpha Builds 1-6 were not publicly available