Go programming - Learning HTTP requestsmultiple threaded program vb.netParsing a page's source code for URLsBouncing ball programLong running, frequent async advice?Multithreading pthread programIs this program susceptible to a buffer overflow exploit?cat program in GoGolang Tour Web Crawler ExerciseDiff for successive sorted sliceCLI Utility Go - First Go Program
Why does a 97 / 92 key piano exist by Bösendorfer?
Air travel with refrigerated insulin
Can I run 125khz RF circuit on a breadboard?
Confusion over Hunter with Crossbow Expert and Giant Killer
Deciphering cause of death?
Personal or impersonal in a technical resume
What should be the ideal length of sentences in a blog post for ease of reading?
Difference between shutdown options
Is there a reason to prefer HFS+ over APFS for disk images in High Sierra and/or Mojave?
How do I fix the group tension caused by my character stealing and possibly killing without provocation?
Pre-Employment Background Check With Consent For Future Checks
Origin of pigs as a species
What is the meaning of "You've never met a graph you didn't like?"
Why is participating in the European Parliamentary elections used as a threat?
Make a Bowl of Alphabet Soup
How to preserve electronics (computers, iPads and phones) for hundreds of years
Telemetry for feature health
Check if object is null and return null
Should I warn a new PhD Student?
Giving feedback to someone without sounding prejudiced
How to make money from a browser who sees 5 seconds into the future of any web page?
What in this world is she trying to say?
Cumulative Sum using Java 8 stream API
Language involving irrational number is not a CFL
Go programming - Learning HTTP requests
multiple threaded program vb.netParsing a page's source code for URLsBouncing ball programLong running, frequent async advice?Multithreading pthread programIs this program susceptible to a buffer overflow exploit?cat program in GoGolang Tour Web Crawler ExerciseDiff for successive sorted sliceCLI Utility Go - First Go Program
$begingroup$
I'm beginning to learn about Golang and I would like to have some advice about the following program.
package main
import (
"fmt"
"net/http"
"time"
)
const BenchmarkTry = 1000
type PageBenchmark struct
url string
time int64 // microseconds
func execBenchmark(url string, channel chan PageBenchmark)
totalExecTimeChan := make(chan int64, BenchmarkTry) // set size to prevent blocked goroutine
totalExecTime := int64(0)
// start all the goroutines
for i := 0; i < BenchmarkTry; i++
go execHttpRequest(url, totalExecTimeChan)
// catch new values from totalExecTimeChan values come from execHttpRequest()) and add it to the total
for i := 0; i < BenchmarkTry; i++
totalExecTime += <-totalExecTimeChan // waiting to get a value from one of the goroutines started in the previous for loop
channel <- PageBenchmarkurl, totalExecTime / BenchmarkTry
// exec http request and attach exec time to channel
func execHttpRequest(url string, channel chan int64)
begin := time.Now()
_, _ = http.Get(url)
channel <- time.Since(begin).Nanoseconds() / 1000000 // convert to milliseconds
func main()
sites := [...]string
pages := [...]string
benchmarkChan := make(chan PageBenchmark, len(sites)*len(pages)) // set size to prevent blocked goroutine
begin := time.Now()
fmt.Println("Beginning !")
// start all the goroutines
for site := range sites
for page := range pages
go execBenchmark(sites[site]+pages[page], benchmarkChan)
// catch new values from benchmarkChan and "print" the PageBenchmark
for i := 0; i < len(sites)*len(pages); i++
benchmark := <-benchmarkChan
fmt.Printf("Url : %vnResponse Time : %d msnn", benchmark.url, benchmark.time)
// print execution time
fmt.Println("End.")
fmt.Println(fmt.Sprintf("%d ms", time.Since(begin).Nanoseconds()/1000000))
So basically here, I'm making HTTP requests (GET method) to multiple urls on multiple web sites. 1000 requests by url in this example.
For now, I just want to start some goroutines in this order :
(First goroutine) mainroutine : start a benchmark (goroutine) foreach web site pages, get the average execution time and print it
(Second goroutine) page routine : start 1000 goroutines (benchmark tries), get the execution times from a channel and store the average execution time in an other channel
(Third goroutine) Execute an HTTP request on the page and store the execution time in a channel
Actually, this piece of code works but I'm not sure that it works as intended.
Is that the case ?
Is the execution scheduling valid ?
Is it appropriate to use defined size channels here ?
Is there a better/more effective way to achieve this task ?
I would be very grateful if you would please take some time to read this topic. Thank you and have a nice day.
multithreading go http benchmarking
New contributor
$endgroup$
add a comment |
$begingroup$
I'm beginning to learn about Golang and I would like to have some advice about the following program.
package main
import (
"fmt"
"net/http"
"time"
)
const BenchmarkTry = 1000
type PageBenchmark struct
url string
time int64 // microseconds
func execBenchmark(url string, channel chan PageBenchmark)
totalExecTimeChan := make(chan int64, BenchmarkTry) // set size to prevent blocked goroutine
totalExecTime := int64(0)
// start all the goroutines
for i := 0; i < BenchmarkTry; i++
go execHttpRequest(url, totalExecTimeChan)
// catch new values from totalExecTimeChan values come from execHttpRequest()) and add it to the total
for i := 0; i < BenchmarkTry; i++
totalExecTime += <-totalExecTimeChan // waiting to get a value from one of the goroutines started in the previous for loop
channel <- PageBenchmarkurl, totalExecTime / BenchmarkTry
// exec http request and attach exec time to channel
func execHttpRequest(url string, channel chan int64)
begin := time.Now()
_, _ = http.Get(url)
channel <- time.Since(begin).Nanoseconds() / 1000000 // convert to milliseconds
func main()
sites := [...]string
pages := [...]string
benchmarkChan := make(chan PageBenchmark, len(sites)*len(pages)) // set size to prevent blocked goroutine
begin := time.Now()
fmt.Println("Beginning !")
// start all the goroutines
for site := range sites
for page := range pages
go execBenchmark(sites[site]+pages[page], benchmarkChan)
// catch new values from benchmarkChan and "print" the PageBenchmark
for i := 0; i < len(sites)*len(pages); i++
benchmark := <-benchmarkChan
fmt.Printf("Url : %vnResponse Time : %d msnn", benchmark.url, benchmark.time)
// print execution time
fmt.Println("End.")
fmt.Println(fmt.Sprintf("%d ms", time.Since(begin).Nanoseconds()/1000000))
So basically here, I'm making HTTP requests (GET method) to multiple urls on multiple web sites. 1000 requests by url in this example.
For now, I just want to start some goroutines in this order :
(First goroutine) mainroutine : start a benchmark (goroutine) foreach web site pages, get the average execution time and print it
(Second goroutine) page routine : start 1000 goroutines (benchmark tries), get the execution times from a channel and store the average execution time in an other channel
(Third goroutine) Execute an HTTP request on the page and store the execution time in a channel
Actually, this piece of code works but I'm not sure that it works as intended.
Is that the case ?
Is the execution scheduling valid ?
Is it appropriate to use defined size channels here ?
Is there a better/more effective way to achieve this task ?
I would be very grateful if you would please take some time to read this topic. Thank you and have a nice day.
multithreading go http benchmarking
New contributor
$endgroup$
add a comment |
$begingroup$
I'm beginning to learn about Golang and I would like to have some advice about the following program.
package main
import (
"fmt"
"net/http"
"time"
)
const BenchmarkTry = 1000
type PageBenchmark struct
url string
time int64 // microseconds
func execBenchmark(url string, channel chan PageBenchmark)
totalExecTimeChan := make(chan int64, BenchmarkTry) // set size to prevent blocked goroutine
totalExecTime := int64(0)
// start all the goroutines
for i := 0; i < BenchmarkTry; i++
go execHttpRequest(url, totalExecTimeChan)
// catch new values from totalExecTimeChan values come from execHttpRequest()) and add it to the total
for i := 0; i < BenchmarkTry; i++
totalExecTime += <-totalExecTimeChan // waiting to get a value from one of the goroutines started in the previous for loop
channel <- PageBenchmarkurl, totalExecTime / BenchmarkTry
// exec http request and attach exec time to channel
func execHttpRequest(url string, channel chan int64)
begin := time.Now()
_, _ = http.Get(url)
channel <- time.Since(begin).Nanoseconds() / 1000000 // convert to milliseconds
func main()
sites := [...]string
pages := [...]string
benchmarkChan := make(chan PageBenchmark, len(sites)*len(pages)) // set size to prevent blocked goroutine
begin := time.Now()
fmt.Println("Beginning !")
// start all the goroutines
for site := range sites
for page := range pages
go execBenchmark(sites[site]+pages[page], benchmarkChan)
// catch new values from benchmarkChan and "print" the PageBenchmark
for i := 0; i < len(sites)*len(pages); i++
benchmark := <-benchmarkChan
fmt.Printf("Url : %vnResponse Time : %d msnn", benchmark.url, benchmark.time)
// print execution time
fmt.Println("End.")
fmt.Println(fmt.Sprintf("%d ms", time.Since(begin).Nanoseconds()/1000000))
So basically here, I'm making HTTP requests (GET method) to multiple urls on multiple web sites. 1000 requests by url in this example.
For now, I just want to start some goroutines in this order :
(First goroutine) mainroutine : start a benchmark (goroutine) foreach web site pages, get the average execution time and print it
(Second goroutine) page routine : start 1000 goroutines (benchmark tries), get the execution times from a channel and store the average execution time in an other channel
(Third goroutine) Execute an HTTP request on the page and store the execution time in a channel
Actually, this piece of code works but I'm not sure that it works as intended.
Is that the case ?
Is the execution scheduling valid ?
Is it appropriate to use defined size channels here ?
Is there a better/more effective way to achieve this task ?
I would be very grateful if you would please take some time to read this topic. Thank you and have a nice day.
multithreading go http benchmarking
New contributor
$endgroup$
I'm beginning to learn about Golang and I would like to have some advice about the following program.
package main
import (
"fmt"
"net/http"
"time"
)
const BenchmarkTry = 1000
type PageBenchmark struct
url string
time int64 // microseconds
func execBenchmark(url string, channel chan PageBenchmark)
totalExecTimeChan := make(chan int64, BenchmarkTry) // set size to prevent blocked goroutine
totalExecTime := int64(0)
// start all the goroutines
for i := 0; i < BenchmarkTry; i++
go execHttpRequest(url, totalExecTimeChan)
// catch new values from totalExecTimeChan values come from execHttpRequest()) and add it to the total
for i := 0; i < BenchmarkTry; i++
totalExecTime += <-totalExecTimeChan // waiting to get a value from one of the goroutines started in the previous for loop
channel <- PageBenchmarkurl, totalExecTime / BenchmarkTry
// exec http request and attach exec time to channel
func execHttpRequest(url string, channel chan int64)
begin := time.Now()
_, _ = http.Get(url)
channel <- time.Since(begin).Nanoseconds() / 1000000 // convert to milliseconds
func main()
sites := [...]string
pages := [...]string
benchmarkChan := make(chan PageBenchmark, len(sites)*len(pages)) // set size to prevent blocked goroutine
begin := time.Now()
fmt.Println("Beginning !")
// start all the goroutines
for site := range sites
for page := range pages
go execBenchmark(sites[site]+pages[page], benchmarkChan)
// catch new values from benchmarkChan and "print" the PageBenchmark
for i := 0; i < len(sites)*len(pages); i++
benchmark := <-benchmarkChan
fmt.Printf("Url : %vnResponse Time : %d msnn", benchmark.url, benchmark.time)
// print execution time
fmt.Println("End.")
fmt.Println(fmt.Sprintf("%d ms", time.Since(begin).Nanoseconds()/1000000))
So basically here, I'm making HTTP requests (GET method) to multiple urls on multiple web sites. 1000 requests by url in this example.
For now, I just want to start some goroutines in this order :
(First goroutine) mainroutine : start a benchmark (goroutine) foreach web site pages, get the average execution time and print it
(Second goroutine) page routine : start 1000 goroutines (benchmark tries), get the execution times from a channel and store the average execution time in an other channel
(Third goroutine) Execute an HTTP request on the page and store the execution time in a channel
Actually, this piece of code works but I'm not sure that it works as intended.
Is that the case ?
Is the execution scheduling valid ?
Is it appropriate to use defined size channels here ?
Is there a better/more effective way to achieve this task ?
I would be very grateful if you would please take some time to read this topic. Thank you and have a nice day.
multithreading go http benchmarking
multithreading go http benchmarking
New contributor
New contributor
edited 7 mins ago
Alex
563210
563210
New contributor
asked 1 hour ago
hunominahunomina
1
1
New contributor
New contributor
add a comment |
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function ()
return StackExchange.using("mathjaxEditing", function ()
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["\$", "\$"]]);
);
);
, "mathjax-editing");
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "196"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
hunomina is a new contributor. Be nice, and check out our Code of Conduct.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
var $window = $(window),
onScroll = function(e)
var $elem = $('.new-login-left'),
docViewTop = $window.scrollTop(),
docViewBottom = docViewTop + $window.height(),
elemTop = $elem.offset().top,
elemBottom = elemTop + $elem.height();
if ((docViewTop elemBottom))
StackExchange.using('gps', function() StackExchange.gps.track('embedded_signup_form.view', location: 'question_page' ); );
$window.unbind('scroll', onScroll);
;
$window.on('scroll', onScroll);
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fcodereview.stackexchange.com%2fquestions%2f215883%2fgo-programming-learning-http-requests%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
hunomina is a new contributor. Be nice, and check out our Code of Conduct.
hunomina is a new contributor. Be nice, and check out our Code of Conduct.
hunomina is a new contributor. Be nice, and check out our Code of Conduct.
hunomina is a new contributor. Be nice, and check out our Code of Conduct.
Thanks for contributing an answer to Code Review Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
var $window = $(window),
onScroll = function(e)
var $elem = $('.new-login-left'),
docViewTop = $window.scrollTop(),
docViewBottom = docViewTop + $window.height(),
elemTop = $elem.offset().top,
elemBottom = elemTop + $elem.height();
if ((docViewTop elemBottom))
StackExchange.using('gps', function() StackExchange.gps.track('embedded_signup_form.view', location: 'question_page' ); );
$window.unbind('scroll', onScroll);
;
$window.on('scroll', onScroll);
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fcodereview.stackexchange.com%2fquestions%2f215883%2fgo-programming-learning-http-requests%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
var $window = $(window),
onScroll = function(e)
var $elem = $('.new-login-left'),
docViewTop = $window.scrollTop(),
docViewBottom = docViewTop + $window.height(),
elemTop = $elem.offset().top,
elemBottom = elemTop + $elem.height();
if ((docViewTop elemBottom))
StackExchange.using('gps', function() StackExchange.gps.track('embedded_signup_form.view', location: 'question_page' ); );
$window.unbind('scroll', onScroll);
;
$window.on('scroll', onScroll);
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
var $window = $(window),
onScroll = function(e)
var $elem = $('.new-login-left'),
docViewTop = $window.scrollTop(),
docViewBottom = docViewTop + $window.height(),
elemTop = $elem.offset().top,
elemBottom = elemTop + $elem.height();
if ((docViewTop elemBottom))
StackExchange.using('gps', function() StackExchange.gps.track('embedded_signup_form.view', location: 'question_page' ); );
$window.unbind('scroll', onScroll);
;
$window.on('scroll', onScroll);
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
var $window = $(window),
onScroll = function(e)
var $elem = $('.new-login-left'),
docViewTop = $window.scrollTop(),
docViewBottom = docViewTop + $window.height(),
elemTop = $elem.offset().top,
elemBottom = elemTop + $elem.height();
if ((docViewTop elemBottom))
StackExchange.using('gps', function() StackExchange.gps.track('embedded_signup_form.view', location: 'question_page' ); );
$window.unbind('scroll', onScroll);
;
$window.on('scroll', onScroll);
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown