CMPUT 410 Final Project Report

Group 5 aka AroundTheBend

James McNeice, uid: mcneice, lab H02, lecture B1 Marko Grasa, uid: grasa, lab H02, lecture B1 Joel Boulet, uid: boulet, lab H02, lecture B1 Sarah Anderson, uid: sma2, lab H02, lecture B1

Key Project Information

Main site start page: http://70.74.230.76:11250/index.pl Seller start page: http://70.74.230.76:11250/startWebSer.pl Designed to run on Apache server configured with perl cgi Requires the perl modules HTMP::Templates, SOAP::Lite, XML::Simple, JSON, DBI, and DBI::MySql, LWP::UserAgent, SOAP::Transport::HTTP, and CGI Please note: We intend to keep the server operational through the end of the April. However, it is possible that an unforeseen circumstances could disrupt it. If you attempt to access the server and its not online send an email to mcneice@ualberta.ca and it will be brought back up as quickly as possible. The most likely error is that a modem restart was necessary which changed the IP it is hosted on. It is also possible that the ssh connection to labs.cs.ualberta.ca closed making the TA hosted webservices inaccesible. Both of these are easily, and quickly fixed.

Overall Design
We decided to write our project using perl cgi scripts as it was the language each of us had used most for the assignments. We designed our project to use an MVC design pattern using HTML::Templates and CSS to creates pages, perl cgi scripts to populate them, and a MySql database as the backend model. HTML::Templates is a perl module allows you to write a page directly in HTML with variables inserted wherever you would like. The variables are inserted as arbitrary text so templates can be combined allowing for complicated pages to be built up of simpler pages. A key example of this is the def_page template we used throughout our project which set up a page header and centered content space. Other pages inserted their own content into the def_page which ensured pages had a standard feel throughout the project. A perl module, authentication.pm, handled checking the user's credentials and setting up the default page with the correct header based on if the user was not logged on, logged in as a normal user, or logged in as a manager. Most pages thus consist of a perl file and a corresponding content template file. All style aspects are controlled by a central style.css file. When deciding where to host our project we had significant reservations about using the provided VMs due to bad experiences during the assignments. We decided to host our own VM using a members spare computer. Our server was a VMware VM running Apache on Ubuntu. Each group member was given an administrator account as well as SSH access to the server. Apache was configured to use /www as the site root and ~user/public_html was available to run test websites for each member. This ended up working very well for us as the server was consistently stable and we could add

required, or testing, perl modules very easily.

Authentication
User authentication was done using cookies containing session ids. When a user logged in a random unique 128 character session id is generated for them and stored in a cookie. A database entry was created with their user id, time when the session was created, a time for the session to expire, and the IP address associated with the session. When a user visited a page the session ID in their cookie was checked to make sure that it was not expired and consistent with their IP address. If a session was consistent the authentication returned their user information for use by the page, otherwise the cookie was set to expire and the user was redirected to the login page. There is an obvious weakness with this approach where a stolen session cookie can be used by anyone behind the same NAT as the user however we felt that it was secure enough in practice for this project. Without being logged in the index (search page) is viewable and usable however to begin a purchase a user must be logged in.

The Seller Module

The seller module consisted of a few main pages; the index which provided search, enquire where a user could begin a transaction, confirmtransaction where a user could complete a transaction, and transcomplete where a user was returned to after paying through paybuddy. Formatting _format.pm contains displayProducts and countryList which are used to ensure a standard appearence through the website and reduce code duplicated. displayProducts takes an array of hash references containing product data and ensures that prices are formatted with the correct number of decimal places and images either exist or are replaced with a default NoImage image. countryList queries the database for a list of all countries that are currently listed as any products country of origin so that the list of countries used for searching is complete. It also formats the country names so that they are lower case with the first character being uppercase.

Searching The index page was index.pl and content_search.tmpl. The index page defaulting to displaying a search returning all products ordered by newest first. Search template created a sidebar inside the overall content area where the user was able to refine their search further. The main part of the content area was a template for previous and next buttons at the top and bottom of the page with a product list in the middle. A template called product_list established a standard way of displaying a list of products throughout
3

the program. The index page uses _search.pm to perform the search itself. The search is done by building up a query that returns only products matching all of the search terms provided. If the search includes keywords they are split on whitespace and checked for in any category; a keyword matching one or more product attributes is considered to match a product. The basic query selects all of the columns from the product table with a blank where. As the search is built up restrictions are concatenated on. Once all restrictions are in place an order by is concatenated on and the search is run. Index.pl checks what result range is supposed to be displayed, jumps to that point in the query, and extracts them in to an array of hash references which are passed off to displayProducts with the result being displayed using product_list. The default full search:

Recommendation If a search returns no results index.pl checks if the user is set as wanting to receive recommendations (the default) and if they are calls recommend in _search.pm. Recommend works by loosening the constraints of the search in an attempt to find one that gets any results. The loosening works by first removing keywords, starting with the last one and moving to the first, and searching after each is removed. We started by removing keywords as we thought they are like to be the least important to a users search, particularly the keywords that were entered last. If removing keywords

fails to return results the first keyword typed is kept and we remove any maximum price defined. We keep the first keyword as we found ourselves often using it to define the critical part of our search, for example by typing Canada rather than using the drop down list, and because results with no keywords can easily return many results. After max price is removed, minimum price is removed, and finally maximum units available is removed. The reason for removing these is that well easily over restricting a search none of them directly affect whether a product would be relevant for a users purposes. If after all of these are tried results can still not be found the program gives up and provides no recommendation. Since further loosening of the search terms, for example by country of origin or category, are likely to return results that are useless for a particular purposes we decided it was better to not frustrate the user by returning pointless results in these cases. Storing recommendations To allow for improvement of the recommendation system over time a database entry is made after each recommendation. The entry contains a human readable version of what the user searched for and what they were recommended. The recommendation id is then attached to the webpage and kept track of up to the user starting a transaction so that it can be noted if the user buys a product that was recommended. The user receiving a recommendation:

Making a Purchase To make a purchase after finding a product the user clicks on enquire and is directed to enquire.pl. Enquire provides a page with product information for only the product youre enquiring about and a form to place an order. The form uses onchange javascript to calculate the price of your order based on the price/unit when the page was loaded and the number of units you are ordering. Its important to note that this calculated price is not used when placing the order and that only the amount that you entered is provided to later pages to ensure that the user can not edit the page and cheat us. After filling out

the form the user clicks order and a transaction is started. Enquiring about a product:

Transactions After clicking order a transaction is started which reserves the quantity of the product that the user wanted and locks in the price. This is done as our interface with paybuddy does not allow us a last second chance to cancel the order before taking money from the buyer in case the product has run out or the price has changed in the meantime. If the user fails to complete the transaction within 24 hours the transaction is closed as incomplete at the quantity of the product on reserve is returned to general availability. This could be abused by, for example, placing a large order on ethanol at the start of the day to lock in that price for the end of the day. If the price went up before the end of the day they could complete the transaction and resell it at a profit and if it went down just backout of the transaction. Since this would be a major issue with the type of goods moving through the service it would be important to use a payment system that allowed for a last second cancellation based on price shift if this were a real world system. Completing a Transaction After clicking order the user is brought to confirmtransaction.pl. Here they have a last chance to review the product being purchase, the amount they are ordering, and the price. If everything is in line they can click on pay and complete to be brought to paybuddy and pay. Paybuddy returns them to transcomplete.pl which displays the amount the product, the amount they ordered, the amount they paid, and the

authentication code from paybuddy confirming the payment. The transaction confirmation page:

User History A logged in user also has access to their Order History and Pending Orders by clicking on the header links in the top left hand corner of the page. Order History and Pending Orders are provided by user_transactions.pl and user_orders.pl respectively. Options are given to search search by product name as well as by date. To enter the dates we used a javascript function the displays a calendar so they can click on dates to select them rather than having to type them in in a particular format. There are also javascript function, detailed in the management section, in place to allow the user to sort the search by any column, other than image, by clicking on the column heading. For pending orders the user is given the option on each order to complete it by visiting the orders confrimTransaction page. The user is also given the option to delete the pending transaction which clears it from the system and returns the reserved product to the general pool. A users pending orders:

Management Component
The Management Component of the website is centered around management.cgi perl file and management.tmpl. This file handles the generation and filling in for all five management pages. In order to identify which page to display, a pageid parameter is retrieved. If there is no such parameter then the Exhibitors list page is taken as the default view. In addition to the pageid, a search parameter as well as start and end time parameters are read in from the search section of the page. These are parsed and validated before being used in later portions of the script. In addition, for the Change User Privileges page, any change user request that has been submitted is handled before further loading of the page. Session authentication is then performed using the _authenticate.pms authenticate subroutine and only managers are allowed to view the page. Setting up the page The next portion of the file defines the key page components that are needed for the content to be displayed. These take the form of a series of hashes which contain the labels, titles and sql queries for each of the valid pageids. This section is key in establishing the basic content parameters for use in the displaying of the page. This is followed by inputting into the content template what the entries in the page select dropbox will be and which is selected. Next comes the simple customizing of the page by inputting the titles, labels and values for the search fields. Loading the reports

Next comes the loading of the customized report results table templates that layout the format for each of the management report tables. This is naturally followed by querying the mySQL database and performing some formatting of the results depending upon the column and what type of values it is meant to contain and show. Finally the page is assembled by adding in the results table to the content page as well as inputting the title and scripts to the base page template. Then, along with printing the html header and the adding of the content pane to the page template the page is printed out and displayed. Javascript sorting The management sections pages all have as their primary feature the table of results. This is sortable through the use of a 3rd party javascript script from: http:// www.kryogenix.org/code/browser/sorttable/ . This script is easy to use in that, other than including it in the html page, you need to merely state that the table is of the sortable css class. Then it automatically makes the table fully sortable by column. While it has a number of other features, the only other feature used was to make the image columns non-sortable as sorting by images is ill defined. Javascript date picker The Search fields for each page include a date picker. The date picker that was used is from: http://www.frequency-decoder.com/2011/10/11/unobtrusive-accessibledatepicker-widgit-v6 and is also fairly easy to use with detailed demos at: http:// www.frequency-decoder.com/demo/datePicker/ . The style and type of date picker used by the management pages is very near the default type used in the first demo on the above page. This was done by declaring a text field and passing in its id to the datepickers creation function along with the date format. Beyond this, the top right corner of the content pane has a drop down box that selects which management page the user wants to view. SQL Queries The Sql queries that were used were as follows: For the Exhibitors report:
SELECT u.user_name as 'Username', u.name as 'Exhibitor', u.date_added as 'Time', COUNT(p.product_Code) as 'Offerings', SUM(p.quantity_avail) as 'AmtAvail', SUM(t.quantity) as 'AmtSold' FROM user u LEFT JOIN product_seller p ON u.id = p.seller_id LEFT JOIN transaction_log t ON u.id = t.seller_id WHERE u.type_seller = TRUE GROUP BY u.id;

For the Buyers report:

SELECT u.user_name as 'Username', u.name as 'Buyer', u.date_added as 'Time',

COUNT(t.id) as 'Purchases', SUM(t.price_per_unit * t.quantity) as 'AmtPaid' FROM user u LEFT JOIN transaction_log t ON u.id = t.buyer_id WHERE u.type_buyer = TRUE GROUP BY u.id;

For the Products report:

SELECT p.product_code as 'ID', p.name as 'Product', p.address_of_picture as 'Image', p.category as 'Category', u.name as 'Exhibitor', p.price_per_unit as 'Price', p.avail_date as 'Time', SUM(t.quantity) as 'Purchases', p.quantity_avail as 'AmtAvail' FROM product_seller p LEFT JOIN transaction_log t ON p.product_code = t.product_code LEFT JOIN user u ON p.seller_id = u.id GROUP BY p.product_code, u.id;

For the Recommendations report:

SELECT r.rec_search_term as 'Recommend', r.inital_search_term as 'Initial', r.date_rec as 'Time', t.recommend_id as 'Result' FROM recommend r LEFT JOIN transaction_log t ON r.id = t.recommend_id GROUP BY r.id, t.recommend_id;

For the User Privileges page:

SELECT u.user_name as 'Username', u.contact_info as 'Contact', u.date_added as 'Time', u.type_seller as 'IsSeller', u.type_buyer as 'IsBuyer', u.type_admin as 'IsAdmin', u.id as 'Id', u.id as 'Submit', u.id as 'FormID' FROM user u"

For All the above queries, as part of a WHERE clause section the search term was put in as a regular expression match against the primary field in the report and a date range inserted in the case of a date range having been requested. The Reports themselves were generated by simply assigning the rows, one by one, to the html table based upon the column title and using looping variables with the HTML::Template perl module. Page Snapshots The Exhibitors report:

10

The Buyers report:

11

The Products report:

The Recommendations report:

12

The User Privileges page:

Web Services
The web services provides the functionality as described in the following document:
13

https://eclass.srv.ualberta.ca/mod/page/view.php?id=184335, that is the: create, update, remove and status methods. Our web services were built with perl and using the SOAP::Lite module. The WebService.cgi file depends on the WebServiceHelper.pm module which contains the following methods: sanitize, getUserID, and checkNecessaryInput. Each web service call first calls the sanitize method to ensure that the input to the functions is safe to process, ie no injection attacks or other malicious attempts on the site, this was necessary due to the site being hosted on an actual server. If an attack is detected the web service will die and return an error message stating the call failed due to a sanitization error. In a production environment the sanitization method should not return anything, but due to this being a learning exercise for both our group and for others we decided to include the return for debugging purpose in the case another group did use a invalid special character they can know why it failed. Externally Accessible Functions The next method that is called by any of the web services is the getUserID which checks to make sure that the password and username provided in the web call are valid and returns a user id for the username since the product_seller table reference the user table in our database. The checkNecessaryInput method was build for the create and update service calls. This class checks to ensure that the necessary parameters that are passed to the methods are valid and not null. Our site requires that the name, country, price, amount, expires and category is not null and that each of those had a valid value. For example, price & amount could not be less than 0 and the category had to be one of: biodiesel, ethanol, feedstock, industrial supplies, equipment and instruments, or other. Our group decided not to implement the remove function to change the expires date to the time of when the web service was called. We did this so that we may datamine our product_seller table to build a better recommendation system as well as for the management module to be able to display all the sales that are and were on the site. The other web service calls function from reading the specifications. The exception to this is that if an error occurred our web services would return an error string starting with Error: and a detailed message stating what went wrong. Functional Limitations It is worth noting that only a user who created an entry in our product_seller table had the rights to remove and update the sale of the item while any user with an account on our site could call the status method on any item. In order for someone to of used our web services he/she required an account on our market. This is unfortunate as it means that a seller wishing to post to all markets needs to visit each market individually and register for an account there rather than allowing a central registration with the market listing. Seller Web Service Interface The web service interface for a seller was built as a standalone unit since it did not share a model with our database instead only communicating through webservices.

14

Please note that if an error page appears in this section it is likely that the ssh connection from the server to the server to labs.cs.ualberta.ca closed so that TA webservices are no longer available. In this case see the note at the top of the page to have the connection restarted. Anyone may use the seller interface even if he/she does not have a registered account on our site, but in order to make a sale on our market one would need an account since the web services require a username and password. The main page for this site is at startWebSer.pl. The homepage is a list of options for the user to create, update, remove or get the status of a sale. Each page requires a user to enter a username and a password. This password and username is the one used as the arguments to be passed to the web service call and is not a login to our market. Since there was no specification set on what groups required as valid input to their market we left the burden of validating to be a server side issue. Create new product The create page asks the user to enter the necessary credentials and to fill out the form. A user may choose to sell on one market at a time or he/she may select All Markets from the dropdown menu to create a sale on every market. Once the user fills out the form they needs to press the go button and the page refresh itself showing a table of returned results from either one or all the markets. The page displays the market name, address of the market, and the message returned from the web service(s). Update a product The update listing page allows the user to fill out a form to update their product listing. Once the user fills out the form and hits go, the page will display if the item was successfully updated or if there was an error in the update process. If there was an error than the page will display the return message from the server- assuming there is one, however since this was not required by the web service specification it may be blank Product Status The status page asks the user for a password, username, a market, and an exhibit Id. If the user chose only one market than a detailed status page will be shown with the status of the product. If all markets were selected a table will show and it will display the market name, market url, and the returned status from the server. Note that although we did allow the user to select all markets for this function (as well as the other ones) it does not make sense in this context since the odds of a user having the same exhibit

15

ids on two or more markets, little alone all the markets, is very low, but we left it build in just to see how it would work. time. Remove a product The remove listing page is the same as the status page except for instead of calling the status method of a web service it instead will call the remove method. Once the user fills out the form and click go the user will see a page stating whether the call was a success or an error was returned. If the user selected all markets instead a table view will show just like in all the other pages, but once again using this function for all markets is unlikely to be helpful as markets set their own ids under the specification.

Database (Model) Design

When we designed the database we wanted something that would meet our needs but was still easy to implement. We were also conscious of the possibility of expansion in the future, thus, we wanted something that could easily grow. To that end our, database has 5 tables in it: product_seller, recommend, session_log, transaction_log, and user. Each of these tables represent an important part of the web site. Tables User User represented the user's who would use the website. This includes buyers, sellers, and admin/management. Each member has a auto-generated id and this was used as the primary key for the table. We tried to fill the table with columns you would usually see on a website/ store database; thus, a user had name, address information, and the type of user the member was. Basically, a persons contact information. This was needed for any transactions to take place. User allowed three types of users, buyers, sellers, and admin. A feature we employed was allowing our members to be both a seller and a buyer. Our group felt that this more reflected a real situation and a users desires. The third type, admin, allowed us a way to specify administrative privileges on the site. Products_seller (products) Product_seller represented the products for sale linked to the user who put them up for sale. Thus, this table had a foreign key to user with strong participation constraints between them. This makes sense because a product for sale shouldnt exist without a seller. Whatmore, by including the seller relationship with product, our table became much more dynamic and easier to maintain. The primary key of product_seller was product_code; this allowed us to uniquely identify each product our website hosted. Another advantage to making this the primary key 16

was it allowed product code searching to be very quick and relatively easy. To improve general searching, the table also contained common searchable fields for columns such as price, unit, weight, quantity and dimensions. This table also include three different types of dates: the date the product was available for selling, the date the product was listed, and the date the product was no longer listed. We felt that by adding these different dates we would give users and management better information and experience. By allowing availability dates, we enabled sellers to advertise their products a few days before they were available. By having a date for when the listing was added, it allows management to keep track of older listings. Having a expiry date allowed us to keep product information in the database. While we never planned to list expired listings on the website, we did want a history of transactions, products, and financials. Having an expiry date gave us this history. Session_log Our third table is session_log. This keeps information about each session a user has on the website. Every time a user logs into the site, a session is started for them. Session_log allows us to keep track of the many sessions that could occur. The table gives information like when the session started, ended, which member initialized it and what ip_address they used to access the site. To keep track of the many possible session logs, the table uses a random generating string as the primary key. Like product_seller, session_log has a participation constraint to user; again, a session should never exist without a user. Transaction_log Our fourth table, transaction_log keeps information about all transactions that have taken place on the website. We store user ids of the buyer and seller, what products were exchanged, and for how much. This allows us to have a save our past and current financial data with the ability to easily calculate statistics about sales, product stock, ect.This table also lists information about recommendations. It allows to determine if a recommendation was used in the transactions. Recommend Our final table is recommend. It stores information about recommendations made to users. We keep track of the initial search term, the recommendation search term and whether the recommendation was taken or not. We also reference a the user who made the search in the first place. This allows us to gather and improve our recommendations.

17

ER diagram for our database:

18