Scalable designs of wide-area upload applications and distributed continuous media systems

Rapid advances in digital signal processing, data compression techniques, commodity computer systems and high speed communication networks have made many data intensive applications across the Internet (such as multimedia applications, web downloads, software distribution) an active research area. In this dissertation, our research focuses on following two main areas: (1) Bistro: a Scalable and Secure Infrastructure for Wide-area Uploads. Hotspots are a major obstacle to achieving scalability in the Internet; they are usually caused by either high demand for some data or high demand for a certain service. At the application layer, hotspot problems have traditionally been dealt with using some combination of increasing capacity, spreading the load over time and/or space, and changing the workload. Some examples of these are data replication (web caching, ftp mirroring), data replacement (multi-resolution images, video), service replication (DNS lookup, Network Time Protocol), and server push (news or software distribution). These classes of solutions have been studied in the context of applications using the following types of communication: (a) one-to-many (data travels primarily from a server to multiple clients, e.g., web download, software distribution, video-on-demand); (b) many-to-many (data travels between multiple clients, through either a centralized or a distributed server, e.g., chat rooms, video conferencing); and (c) one-to-one (data travels between two clients, e.g., e-mail, e-talk). However, to the best of our knowledge there is no existing work, except ours, on making applications using many-to-one communication scalable and efficient at the application layer; existing solutions, such as web based uploads, simply use many independent one-to-one transfers. This corresponds to an important class of applications, whose examples include the various upload applications such as submission of income tax forms to IRS, conference paper submission, proposal submission through the NSF FastLane, homework and project submissions in distance education, Internet-based storage, and many more. The main focus of our work is scalable infrastructure design for wide-area upload applications. This area encompasses a broad range of challenging topics including (a) the basic framework and general solution for wide-area upload applications, (b) server selection problems, (c) providing for security needs of upload applications, (d) large-scale data collection, (e) cost-effective resource management and allocation techniques, (f) provision of fault tolerance schemes to ensure that all data arrives at the destination server in the face of failures or malicious behavior by the hosts or elsewhere in the network, and so on. My dissertation addresses topics (a)–(d). Remaining topics are part of ongoing and future efforts. (2)  Continuous Media Servers. Multimedia applications (such as video stream delivery, digital libraries, and distance learning systems) place high demands for quality-of-service (QoS), performance, and reliability on storage servers and communication networks. These, often stringent, requirements make design of cost-effective and scalable continuous media (CM) servers difficult. Therefore, the main focus of my work is to provide an efficient adaptive and dynamic resource management techniques in conjunction with data placement techniques in order to improve performance, scalability and reliability of such systems. In this work, we propose to use dynamic replication techniques in conjunction with a “hybrid data placement” technique to design a scalable CM server. Through a performance study, we show that not only does this hybrid design with dynamic replication techniques result in a scalable, cost-effective CM server, but it also suggests the usefulness of these techniques across a wide range of continuous media applications.